As every developer knows, what works in the ideal scenario (a.k.a. your production environment) won’t necessarily work in the real world. That’s why you need tools like Retrace to help you quickly track down bugs and performance issues to keep things running smoothly.
Load testing, just one of many valuable testing methods you can use before release to ensure that your application is bug-free and runs seamlessly, is the process of testing whether an application can withstand the load of concurrent users, requests, and data volumes when in use. In other words, load testing is a must for determining if your applications can stand up to real-world use conditions.
There are many load testing tools available today for all kinds of applications and organization. In this article, we’ll look at 50 different load testing tools and highlight the key features of each of them.
Note: The following 50 load testing tools are not ranked or rated in any particular order of importance, but are listed in random order below.
1. Apache JMeter
Apache JMeter is a 100% pure Java, Open Source application that can be used to test performance on both static and dynamic resources, simulating heavy loads on individual servers, groups of servers, networks, or objects to test strength and analyze performance under different load types. Check out the getting started documentation here.
Key Features:
- JMeter is a Java application for load and performance testing
- Used to test web apps, SOAP & REST web services, FTP, databases, and more
- The most mature, and widely used open source load testing tool available
- Many commercial tools support JMeter
Cost: Open Source
2. WebLOAD from RadView
WebLOAD from RadView powers the most demanding performance professionals in the world with enterprise-scale load testing.
Key Features:
- Lets you record actions and produces a test script in JavaScript
- Visually intuitive interface makes it easy to create and manage load testing scripts
- Enables performance testing for web, server-side, cloud, and mobile applications
- Well integrated with enterprise products from Oracle, SAP, Microsoft, and more
Cost:
- Free Edition available
- Contact for a quote
3. LoadComplete from SmartBear
A desktop tool for load, stress, and scalability testing for both websites and web apps, LoadComplete by SmartBear doesn’t require advanced coding skills – meaning you can focus on the more important tasks. Check out LoadComplete’s documentation here.
Key Features:
- Load testing for .NET and Java apps
- Generates various levels of traffic without requiring code inputs
- Test options available for IE, Chrome and Firefox
- Provides checks for both, secure and non-secure traffic
Cost: Plans start at $230 – $1,625 per month, based on a 1-year subscription
4. NeoLoad
A load and performance testing tool that realistically simulates user activity, NeoLoad helps you eliminate bottlenecks in your web and mobile apps. Check out the documentation here.
Key Features:
- Supports Selenium script conversion for Selenium C# developers
- Provides ‘drag and drop’ loops, conditions and other controls
- Checks user paths to find newer path versions to avoid errors
- Makes recording faster by providing prefilled transaction list
Cost: Contact for a quote
5. Locust
Locust is an Open Source load testing tool that enables you to define user behavior using Python to test your system’s performance under millions of simulated, simultaneous users. Full Locust documentation is available here.
Key Features:
- Allows you to write expressive scenarios in plain-old Python
- Enables testing on any system, and multiple systems at the same time
- Uses community members to add to the traffic
Cost: Free
6. BlazeMeter
A performance engineering platform for DevOps, BlazeMeter enables you to run and analyze JMeter and other open-source load tests from anywhere. Check out the documentation here.
Key Features:
- Quickly pinpoints defects using inbuilt error report function
- Detailed and interactive report timeline graph to make the report clearer
- Thorough testing using both, API backend and frontend
- Uses YAML and JSON syntax for developers using these languages
Cost:
Andaaz - Akshay Kumar, Priyanka Chopra, Lara Dutta. Dil Kare Chu Che - Singh Is Bliing. Find out where to watch, buy, and rent Andaaz Online on Moviefone. After his sweetheart Kajal marries another, Raj moves to Europe. He eventually agrees to wed Jiya but is. Andaaz is a 2003 Indian romantic musical film directed by Raj Kanwar. Chu Che - Singh Is Bliing. Andaaz watch online.
- Free for 50 concurrent users, 10 tests, 1 shared load generator
- Plans start at $99 – $499 per month
7. Load Multiplier
Load Multiplier is a testing framework to test IT products across domains such as telecommunication, banking, web, proprietary protocols. Additionally given a client library, it can generate millions of such client instances so as to generate real field traffic. It comes with a web based front end, and powerful C/C++ based back end engine.
Key Features:
It provides stable uninterrupted long hour load testing solution for products based on:
1. WebRTC
2. IoT
3. SIP, IMS
4. RTP, RTCP
5. STUN, TURN, ICE
6. HTTP, SOAP, REST
7. JSON, XML
8. G711u, G711a, AMR, OPUS, VP8
9. Any proprietary text / binary protocol
Cost: Contact for a quote
8. WAPT
A load and stress testing tool for easily analyzing website performance, WAPT creates tests in minutes for mobile applications, web services or customized ERP systems. The full WAPT documentation is available here.
Key Features:
- Sends reports to mobile device and desktop device for convenience
- Provides extensions to work with ASP.net, Adobe Flash and Silverlight
- Test results are shown in 20 different detailed reports
- Provides detailed graphs even during the runtime, to help track the performance
Cost:
- WAPT: Starts at $700
- WAPT Pro: Starts at $1,200
9. Test Studio from Telerik
A complete test automation solution for GUI, performance, load, and API testing, Test Studio from Telerik is a useful tool for testing desktop, web, and mobile applications. The full documentation is available here.
Key Features:
- Provides testing for HTML5, AJAX, Android, WPF
- Allows users to share and pick references from Element Explorer
- Developers can use any language they are used to
- Users can test any NativeScript app over 500+ devices
Cost:
- DevCraft: $1,499/developer (perpetual license)
- Telerik Platform: Starts at $39 per month per developer
10. Loader.io
Loader.io offers simple cloud-based load testing for stress testing your web apps and APIs with thousands of concurrent connections. Check out the documentation here.
Key Features:
- Runs tests for you anytime and from anywhere
- Allows monitoring of the test and performance from any place
- Provides stress test result in descriptive graphs which will be sent to all users
Cost:
- Free: 10k clients/test
- Pro: $99.95 per month – 100,000 clients/test
11. HPE LoadRunner
Get a complete picture of end-to-end system performance to identify and resolve issues before your apps are released with HPE LoadRunner. Check out the LoadRunner documentation here.
Key Features:
- Reduces the time it takes to understand and explain the reports
- Supports wide range of applications to decrease time and skill
- Provides detailed mobile performance test reports
- Combines load testing with development tools like IDE, jUnit and more
Cost: Contact for a quote
12. StormRunner Load from HP
An SaaS-delivered cloud load and performance testing solution, StormRunner Load makes it simple to plan, run, and scale testing for both web and mobile apps. You can read the full documentation here.
Key Features:
- Takes 10 minutes to design and create web load or mobile test
- Tests can be managed from anywhere using a cloud based testing function
- Can quickly scale from 1 to 1,000,000 geographically distributed web and mobile user
Cost: Contact for a quote
13. CloudTest from Soasta
CloudTest provides scalable and affordable load testing tools for more efficient, transparent performance testing. Check out the full CloudTest documentation here.
Key Features:
- Gets real user experience from 95 different locations in all 7 continents
- Provides you with 40 cloud service providers to make your testing seamless
- CloudTest automation tools quickly and easily integrate with yours
- Importing existing JMeter test scripts have been made easy
Cost:
- Lite: Free for 100 virtual users
- On-Demand: Starts at $2,500
14. The Grinder
A Java load testing framework, The Grinder makes running distributed tests simple using many load injector machines. The Grinder is maintained by Philip Aston, who took over the code (originally developed for the book Professional Java 2 Enterprise Edition with BEA WebLogic Server by Paco Gómez and Peter Zadrozny) and reworked it to create The Grinder 2 and subsequently The Grinder 3. The full documentation can be found here.
Key Features:
- Test scripts can be written in Clojure and Jython
- Any Java API can be loaded as a test, be it from HTTP web servers, SOAP and REST services, etc
- Pre-built Java libraries with large variety of systems and protocols is available
Cost: Free
15. Tsung
An Open Source, multi-protocol, distributed load testing tool, Tsung helps you test the performance and scalability of IP-based client/server applications. Check out the full documentation here.
Key Features:
- Multiple IP addresses can be opened on one machine using their OS IP Aliasing
- Response time can be measured during the load by generating HTML reports
- HTTP, WebDAV, MySQL and LDAP servers can stressed using this software
Cost: Free
16. Gatling.io
An Open Source load and performance testing tool for web applications, Gatling helps you improve time-to-market by identifying bottlenecks, errors, and performance issues prior to release. Gatling documentation is available here.
Key Features:
- Can forecast performance issues in the web application you make
- Errors and bottlenecks will be detected in the early part of the development cycle
- Drastically reduces the debugging phase time
- Provides the best digital experience to customers and users
Cost: Free
17. Taurus
Because automating repetitive tasks just makes sense, Taurus provides an automation-friendly framework for continuous testing. You can read the full documentation here.
Key Features:
- Provides smooth automation and integration of JMeter and other systems
- Creating, running and analyzing performance tests are made simple
- Has a different approach and system than The Grinder and Gatling
- New tests can be made from scratch using control friendly and unified DSL
Cost: Free
18. Httperf
Httperf is an Open Source HTTP load generator for measuring web server performance that facilitates the construction of both macro- and micro-level benchmarks.
Key Features:
- This is a simple open source tool for measuring web server performance
- It is solely made to test standard http payload of the application you test
- High performance and robust tool that helps create micro and macro benchmarks
- It is able to create and sustain server overloads
Cost: Free
19. Silk Performer from Micro Focus
Micro Focus Silk Performer understands that the biggest and most complicated issues happen at the most inconvenient times, so Silk Performer is designed to provide a consistent user experience anywhere, anytime, on any device. Check out the documentation here.
Key Features:
- Load tests reflect real world user behavior with their different constraints
- Provides unlimited cloud support for load testing on mobile devices and systems
- Prepares application for low bandwidth users
Cost: Contact for a quote
20. ApacheBench
A tool designed for benchmarking an Apache Hypertext Transfer Protocol (HTTP) server, ApacheBench gives you an accurate picture of how your current Apache installation performs. The documentation is available here.
Key Features:
- Provides better support for asynchronous read and write
- Keepalivetimeout can be specified in a millisecond
- Helps you understand how your current Apache installation performs
Cost: Free
21. ZebraTester
Offering powerful tools and flexible options, ZebraTester enables you to easily create and run tests of nearly unlimited scale with incredible visibility into your results. Check out ZebraTester’s documentation here.
Key Features:
- Includes performance testing for CI/CD process using LoadTest Portal
- Apica team helps design, execute and manage the project
- Security policies have been given equal importance, and subsidiary software is available
- Free 500VU are provided on their global network and LoadTest Portal
Cost: Contact for a quote
22. Experitest Mobile Add-On for LoadRunner
Experitest offers several testing tools for mobile apps, including a mobile add-on for HP LoadRunner, which makes it possible to perform mobile load testing by connecting real devices and testing your applications under simulated loads. Read the full documentation here.
Key Features:
- This is specially made to test your mobile applications load taking capacity
- Runs the same tests as systems on the mobile devices
- Create and edit mobile tests inside LoadRunner
- Creates report screenshots and videos of the application
Cost: Contact for a quote
23. Load Impact
With nothing to install and easy configuration, Load Impact offers instant load testing for devops and developers. Read the full documentation here.
Key Features:
- Connects you with performance trending analytics and notifies through webhook
- Provides a powerful Lua scripting environment to allow you to create simple or complex API scenarios
- Record results of test uploaded on website or app using their chrome extension
- They have provided more than two million customers with load testing service
Cost: $89 – $1,299 per month
24. AgileLoad
Optimize your load and performance testing costs with AgileLoad, which provides distributed and cloud-based testing on-demand and server and database monitoring in a single package. Documentation for AgileLoad can be found here.
Key Features:
- Helps you create complicated scenarios in minutes using their Web 2.0 technology
- Allows you to simulate your load from any location, through the Cloud
- Gives you an accurate end to end picture of your application’s performance
- Lets you correlate different load test by combining them in one graph
Cost:
- Starts at $192 per month
- Daily rental starting at $24 per day
25. LoadStorm
An easy and cost-effective way to load test both web and mobile apps, LoadStorm helps you find the breaking point of your applications and other weaknesses before your customers do. Check out LoadStorm’s Getting Started documentation here.
Key Features:
- Provide full project management, where they do all the test work for you
- Cloud servers are able to run load tests up to 1,000,000 VUsers
- Allows logins from any location around the world
- Provides control of spreading the traffic to any geographic location
Cost: Contact for a quote
26. Visual Studio Team Services
Visual Studio Team Services offers cloud-based load testing to performance test your code under the weight of thousands of users. Check out the documentation here.
Key Features:
- Uses Azure to make Cloud-based load testing easy and quick
- Visual Studio Team Service can help make tests by your references of other websites
- Correlates test results with server diagnostics
- Functions to record and replay your actions are also available
Cost:
- Free up to 5 users
- Growing Teams: Starts at $30/month for 10 users
27. Loadster
A full-featured testing solution for websites, web apps, and web services, Loadster is built for real web applications and can handle cookies, user sessions, custom headers, and more with ease. Check out Loadster’s documentation here.
Key Features:
- Gives a hybrid solution for load testing with ground server and cloud system
- Provides two options for generating concurrent user, from their server and your own
- Concurrent users are generated from 5 different continents
- Loads multiple tests at once and generates users for all tests
Cost: $79 – $2,999
28. LoadView
LoadView provides flexible load testing from the cloud for websites, web apps, mobile, and APIs. With a 100% managed cloud, there’s no need to set up your own third-party cloud accounts.
Key Features:
- Complete Cloud-based load testing software, eliminating hardware hassle
- Allows you to pay according to the use if you don’t want to pay for a month
- Shows you the load stress for real world browsers
- Trusted by Volvo, Dell and ComCast
Cost (per load test):
- Each Virtual Machine: $4
- HTTP Load: $0.01 per user per minute
- Full Browser Load: $0.10 per user per minute
- Flat monthly cost to store test result data: $9.99
29. Artillery
A modern, powerful, and easy-to-use load testing toolkit, Artillery empowers developers to make scalable, performant, and resilient applications that can stand up to high load. The documentation is available here.
Key Features:
- Supports multiple protocols- HTTP(S), WEbSockets and Socket.io
- Software prefered by ecommerce backends, loT bankends and transactional APIs
- Supports Statsd, Datadog, InfluxDB and Librato
- Use npm modules and customize load tests with Java Script
Cost: Free
30. eggPlant
Ensure that your cloud and server-based applications will continue to work, perform flawlessly, and provide a consistent user experience with eggPlant. Check out eggPlant’s documentation here.
Key Features:
- Technology support includes client-side .NET, client-side Java, Citrix and TCP/UDP
- Load test creating tools are easy, which helps new and experienced users design
- C# and Java are fully integrated to help faster script authoring
- Provides live monitoring to help control tests while they run
Cost: Contact for a quote
31. Applause
Offering a range of load testing services, Applause helps you ensure that your apps and websites can handle the biggest loads on your most successful days. Check out the documentation here.
Key Features:
- Pre-built custom load scripts to reduce scripting time and money
- Provide expert help to interpret load test results for better insights
- Give access to experts who craft custom Apache JMeter load testing scripts
- Provides cloud-based infrastructure for flexible scaling
Cost: Contact for a quote
32. LoadUI
LoadUI provides load testing for REST and SOAP APIs to ensure your APIs never let you down. You can rapidly create API load tests either against a single web service endpoint or an existing functional API test.
Key Features:
- Allows reuse of existing SoapUI Pro functional tests
- Can run several load test scenarios continuously with one another
- See the interaction of each scenario and the impact they have
- Shows how your server reacts to traffic and diagnose the bottlenecks
Cost: $4,999 – $14,999 per year
33. JCrawler
JCrawler is an Open Source stress testing tool for web apps with a crawling/exploratory feature enabling you to define a set of starting URLs for JCrawler to begin crawling and generating loads. Check out the documentation here.
Key Features:
Best Software Automation Test Tools
- Can be used through little bandwidth, doesn’t require high internet speeds
- Uses XML file to configure, which makes it easier to create load tests
- Can be run on any type of system
- JCrawler tools specially made for developers and QAs
Cost: Free
34. StresStimulus
A load testing tool for both web and mobile applications, StresStimulus measures web performance and scalability under heavy traffic loads. The full StresStimulus documentation can be found here.
Key Features:
- Records traffic for mobile devices like Apple, Android, Windows and Blackberry
- Works smoothly with all main web platforms- Java, PHP and ASP.NET
- Supports many Enterprise applications CRM, SharePoint, Silverlight and more
- The only software that can be used as an add-on for
Cost: $1,040 – $3,950 per month (Perpetual 250 VUs License)
35. OpenSTA
A distributed software testing architecture designed around CORBA, OpenSTA performs scripted HTTP and HTTPS heavy load tests with performance measurements from Win32 platforms. The documentation can be found here.
Key Features:
- Made for experienced and proficient testing users
- Records can be played by different users’ perspectives for better understanding
- Open Source tools licensed under GNU General Public License
- Was made to be free for user and will always remain free
Cost: Free
36. SOAtest from Parasoft
With SOAtest from Parasoft, you can automate complete end-to-end testing for both business- and security-critical transactions. Check out the documentation here.
Key Features:
- Provides a user-friendly interface for monitoring live application of traffic
- The tests are engineered to shared, reused and extension easily
- Includes static analysis, code review, code coverage analysis and runtime error detection
- Designed to make load testing without scripting possible
Cost: Contact for a quote
37. APImetrics.io
You can’t afford latency in your APIs, and APImetrics aims to solve that challenge by putting real-time API performance data at your fingertips. APImetrics documentation can be found here.
Key Features:
- Provides real-time alerts and notifications covering failures, speed and size issues
- Daily, weekly, monthly and yearly review given through detailed reports
- Gives you functionality for every sector you service
- Allows you to measure the latency and uptime from any location or cloud
Cost: Starts at $10,000 per year
38. Goad
An AWS Lambda powered, distributed load testing tool, GOAD enables you to launch HTTP loads from up to four AWS regions simultaneously. Check out GOAD’s documentation here.
Key Features:
Brendan Fraser in The Mummy Returns (2001) Brendan Fraser and Arnold Vosloo. See full technical specs ». People who know their action flicks will notice that the Mummy Returns steals about every action scene from every action movie. The Mummy Returns: A Netflix Original. Adventures, Monster Movies, Sci-Fi & Fantasy, Blockbuster Sci-Fi & Fantasy, Action Sci-Fi & Fantasy, Fantasy Movies. The year's biggest action-adventure is coming home with a vengeance! Brendan Fraser and an all-star cast reunite with Stephen Sommers for explosive,. Watch the mummy returns full movie.
- Allows HTTP load launch from four different AWS at the same time
- A single lambda is able to handle hundreds of concurrent connections
- Peak load of up to 100,000 concurrent requests
Cost: Free
39. Reflective
A library injection technique that employs the concept of reflective programming, Reflective DLL injection performs the loading of a library from memory into a host process. Check out the documentation on GitHub.
Key Features:
- This follows the concept where reflective programming is employed
- The library loads itself by using minimal portable execution
- It interacts with the minimal interaction with the host system
- Injection works on Windows NT$ and up including Windows 8
Cost: Free
40. LoadTracer
A user-friendly tool for web application load testing and performance testing, LoadTracer works like many other load testing tools, recording scripts and playback with hundreds of virtual users.
Key Features:
- It helps with web application load testing, monitoring and performance testing
- Uses various techniques to record and find bottlenecks and latency issues
- Also provides tools for website loading tests
Cost: Contact for a quote
41. Solex
A free, Open Source web application tool, Solex was built as a plug-in for Eclipse IDE. The full documentation can be found at SourceForge.
Key Features:
- Provides functions to record client sessions and replay it later after adjusting
- It acts as a HTTP proxy and records HTTP requests between Web client and server
- It is an international Open source project, which makes it more helpful
- It is based on Eclipse platform, making tools fully integrated
Cost: Free
42. Bees with Machine Guns
Bees with Machine Guns is a nifty little utility that sends swarms of “bees” to attack (a.k.a. load test) your web applications. Check out the documentation here.
Key Features:
- Helps create micro EC2 instances called bees to load test web applications
- Uses Python 2.6 – 3.6, Boto, Paramiko dependencies
- Can be pointed at any web application to test the load taking capacities
Cost: Free
43. Appvance
The first unified test automation platform that completely transforms the software QA process, Appvance eliminates silos and helps you see your app’s performance through your users’ eyes.
Key Features:
- Tests your applications and shows how a user would see it
- Finds bugs the user would find on their browsers and mobile applications
- Data-drive is available for any language you use
- Brings in a new level of smoothness in their record and play script creation
Cost: Starts at $99 – $1,200 per month
44. nGrinder
nGrinder is an enterprise-level performance testing solution based on The Grinder, allowing you to execute script creation, test execution, monitoring, and the result report generator simultaneously. Check out the documentation here.
Key Features:
- Uses Jython and Groovy to design scenarios and create stress in JVM
- Tests can be extended to limitless customs, using their unlimited library
- Library consists Jar, Python, Maven dependencies
- Allows you to monitor the agents creating stress and the targeted machines
Cost: Free
45. Rational Performance Tester from IBM
Test earlier and more often with Rational Performance Tester, which validates web and web server applications for scalability, identifies bottlenecks, and more. Check out the documentation here.
Key Features:
- Allows early and frequent tests of DevOps approach
- It helps see the extent of the web and server scalability
- Identifies the presence and cause of the performance bottlenecks and system issues
- Provides root cause analysis feature to find and diagnose the issue
Cost: Contact for a quote
46. J-hawk
An Open Source Java-based framework that you can incorporate into your applications for performance testing, J-hawk generates performance reports to help you identify bottlenecks. Check out the documentation here.
Key Features:
- An Open Source Java based framework to test the performance of the application
- Uses its own scripting language called $hawk scripting
- Provides two ways to use j-Hawk- Property Mode, Script Mode
- j-Hawks solves what junits does not solve
Cost: Free
47. Selenium
Selenium isn’t actually a load testing tool itself, although its powerful browser automation features enable various load testing scenarios for low-scale browser loads. Check out Selenium’s documentation here.
Key Features:
- Though not a load testing tool itself, its powerful browser automation features enable various load testing scenarios
- Provides two types of applications for user, based on the kind of work they want to execute
- Selenium WebDriver creates robust browser tests and automation suites
- Selenium IDE designs scripts to help Automation exploratory test
Cost: Free
Check out our Selenium tutorial!
48. Performance Center from HP
With Performance Center from HP, you can plan and execute tests across multiple global projects. With software testing tools and consistent processes, you can create a world-class testing center with ease. The documentation is available here.
Key Features:
- Is updated to be compatible to all sorts of hardware
- Creates a common testing infrastructure for engineers located around the globe
- Use tests stored in the Cloud or in your premise behind a firewall
- Reduces cost by creating a shared tests report infrastructure for engineers
Cost: Contact for a quote
49. Hammerhead
A web site coverage, HTTP load generator, HTTP benchmarking, and stress testing tool, Hammerhead is designed to emulate several users from different IP addresses at maximum speeds to push your apps to the limit.
Key Features:
- It is a HTTP load generator and benchmarking tools
- Provides web site coverage and website stress testing tools
Cost: Free
50. Load Xen
Load Xen, by Testcollab, enables you to simulate millions of requests to help you more effectively scale your applications. You can create interactions that seem just like real users to your application, randomize requests, run concurrent tests, and more with Load Xen.
Key Features:
- Simulate hundreds to millions of virtual users
- Create interactions that mimic real use
- Flexible scripting
- Real-time reporting
- Concurrent tests
Cost:
- Free: 100 VUs, max test time 2 hours
- Paid plans: $50 to $500 per month
It is important to test any software’s performance and load handling capabilities before it is released. There are many options available ranging from open source to enterprise-ready tools to perform load testing. We hope this list will help you choose the right load testing tool for your project depending on the kind of software you are building and your budget.
- Looking for New Relic Alternatives & Competitors? Learn Why Developers Pick Retrace- May 1, 2019
- Better code, Better results: No BS [Webinar]- December 20, 2018
- New Feature in Retrace: Application Scoring- May 24, 2018
- What’s New in Retrace: Java on Windows Support Now Available- March 20, 2018
- SST Software Fixes Problems behind the Scenes with Retrace- February 1, 2018
Test automation allows software project teams the ability to create repeatable, reusable test scenarios that can be executed as often as needed to drive software quality up and manual labor down.
Though, success in test automation depends on the type of tool chosen. Different considerations matter for mobile vs. web testing. How do you choose? We couldn’t resist an automated software testing tools comparison reviewing 11+ along with a list of questions you can use to help you decide.
When Should You Automate?
Important to note in our automated software testing tools comparison — not everything is possible to automate, and all the scenarios possible aren’t necessarily worth automating. Software project teams choose to invest the upfront time in setting up automation test cases for a few reasons:
1. Reports required. The client/product owner/stakeholders involved require extensive daily/weekly documentation. Using automation, reports can be run repeatedly with no additional workload.
2. Length. The project is expected to span over many months. For a two-month project, the cost to invest time in automation isn’t justified. The level of complexity doesn’t warrant the additional coverage. Now, for a two-year project? Investing in automation upfront can ensure that as features are developed, tested and the team moves on, new functionality can be fully regression tested.
3. Focus. For comparison against test automation, most manual tests are end-to-end. Using automation they are more difficult to maintain and stabilize. That’s why teams who opt to automate break up tests into more digestible pieces keeping the focus on individual or small combinations of functions. That way, when code is updated, only the test automation with related functionality needs to be updated.
How Software Teams Choose an Automation Test Tool
Test automation offers the possibility to validate mobile apps instantly and effectively. Once tests have been automated, they can be executed quickly and repeatedly, again and again. In almost all cases, this is the most cost-effective method for regression testing software products that have a long maintenance life.
In fact, test automation of any mobile app is the best way to increase the effectiveness, efficiency and coverage of the testing process. Generally, teams setting up test automation are focused on selecting a tool that allows them to maximize the following success criteria:
Reliability: High rates of issue detection
Maintainability: 1) Minimum sensitivity to application and test case changes 2) Test case definition separate from automation code
Scalability: The ability to efficiently expand test coverage and automation framework if needed
There are even more considerations once teams make the leap to mobile test automation. The following factors make executing test automation for mobile even more complex:
- Platform diversity
- Device diversity
- Tool diversity
- Tool immaturity
- Rapid development cycle
- Rate of technology change
When selecting a test tool, platform compatibility, flexibility, available support and cost should be top-of-mind. Here’s how we recommend assessing possible tools along each dimension:
Platform Compatibility
Verify the tool’s ability to interact with all supported mobile and desktop platforms.
Functional/Flexible
Check that the tool being considered supports the most important application for your teams (client, admin, etc.)
Available Support
Compare the support teams, features or communities in place to help teams or clients overcome technical hurdles.
Cost of Ownership
Assess the long-term investment purchasing/setting up or supporting the tool requires compared to the expected return.
?’s to Ask in Your Automated Software Testing Tools Comparison
- Does the tool support multiple OS platforms?
- How quickly can the tool support new mobile OS releases?
- How can you use the tool to execute a single test across multiple platforms?
- How does the tool interact with devices remotely?
- How many scripting engines does the tool support?
- Is the tool visual or object-based?
- Does the tool require jailbreaking or rooting?
- Can it be used for automation of web, hybrid and native mobile applications?
- Does it require a code change for automation support?
- How much does the tool cost?
- How much support is offered by the tool?
Web Automation Tools
Selenium IDE
This Firefox plugin provides the ability to record test cases. Best used for testing web apps, functionality is limited and unsuitable for complex web applications. The tool does not allow teams to write tests for flash video games, music, UI/UX tests or file uploads.
Selenium WebDriver
When using Selenium WebDriver, test cases can be written in a wide variety of programming languages including (C#, Java, Ruby, Python, etc.). This open source market leader has a large following and formidable training resources/support articles available. It can be easily integrated with Cucumber/SpecFlow.
Sikuli Script
Sikuli can accommodate recording and testing web and desktop applications. The tool uses image recognition to identify and control GUI components. It is particularly useful when easy access doesn’t exist for a GUI’s internal or source code.
Visual Studio (CodedUI)
Teams can rely on Visual Studio to test web and desktop applications. Again, it is useful when no easy access to GUI’s internal or source code exists. Though, test cases are written in C# and are not easy to maintain.
Mobile Automation Tools
Appium
According to the Appium website, the tool offers “mobile app automation made awesome.” We couldn’t agree more. The open source tool supports iOS and Android applications, along with native and hybrid applications. The test cases can be written in a wide variety of programming languages including (C#, Java, Ruby, Python, etc.). Different design patterns can be used to minimize test cases maintenance. Tests can be integrated with Cucumber/SpecFlow.
Appium is cross platform meaning tests can be written against multiple platforms using the same API.
Pros
- Web, hybrid and native app support
- Good community-based technical documentation
- Supports Android and iOS
- Minimum code change required for automation
- Supports different language scripts
Cons
- Initial setup time
- It can only run one instance per Mac
Many know Appium as the mobile version of Selenium. Because the two tools are so similar, test teams can create one framework for both rather than two. That means if teams are testing in both, there’s much less code to maintain over the course of the product.
Calabash
Calabash is an open-source option that can be used for iOS and Android projects. Test cases are written in Ruby, which can pose a problem for staffing in the lesser known language. Different design patterns can be used to minimize test cases maintenance.
![Automation testing tools list Automation testing tools list](https://www.conversion-uplift.co.uk/wp-content/uploads/2015/03/AB-Testing-Solutions-Matrix-May-2016-1024x677.png)
Robotium
The free tool allows users to record native and hybrid Android tests. It is supported by Google, can be integrated with Cucumber and written in Java.
Desktop Automation Tools
TestStack.White
TestStack.White is a free tool used to support Win32, WinForms, WPF, Silverlight and SWT (Java) applications. Test cases are written in C# and can be integrated with Cucumber/SpecFlow.
SpecFlow
SpecFlow is a free offering that provides behavior driven testing capabilities for languages in the .NET family.
Cucumber
Like SpecFlow, Cucumber offers behavior driven testing capabilities only for Ruby and Java.
Service Automation Tools
SoapUI
The resource offers both open source and paid versions and provides a means to create load and security test cases. It is used for REST API testing, but functionality can be expanded with Groovy scripts.
HttpWebRequest
The HttpWebRequest can be integrated with SpecFlow and Cucumber. It is a .NET library which makes the REST API request directly. Can be easily integrated with Cucumber/SpecFlow.
Additional Consideration: Testing in the Cloud
We would be remiss to exclude cloud testing options from our automated software testing tools comparison. As screen variety increases, testing in the cloud continues to appeal to businesses. Options including Xamarin Test Cloud, Amazon AWS Device Farm, Sauce Labs, Testdroid and SOASTA allow businesses to expand test coverage and test on all possible devices without physical limitations.
Pros
- No growing device lab
- Equally accessible to all team members in a distributed team
- Optimize speed, efficiency and quality
- Increase of test coverage
- Regression testing after bug fixing or further development of software
- Testing of software on different platforms or with different configurations
- Data-driven testing (running same test cases using many different inputs)
- Test automation allows performing different types of testing efficiently and effectively
- Expedite the release cycle by reducing time for regression testing
- Quality improvement
Cons
- Often the application under test must be accessed by a VPN. If the cloud-based device has difficulties accessing the code, it means more setup time for the development team.
Test Automation Success Factors
Planning. Building adequate time into the product lifecycle is a core differentiator for successful projects. Identify where your QA team will fit into the process. Will they test every user story? Or will you attempt a combination risk-based/automation testing strategy? These are all things to consider before the team begins development.
Clarity. Clearly define/document test cases and ensure all members testing understand the requirements/functionality being validated.
Control. Ensure your quality analysts/engineers have dedicated testing environments and full control over test databases.
Timing. Our team has found QAs can test most effectively if they are validating/reporting on code developed one sprint behind the development team. Without the lag, the code would be too fluid to test effectively; changes would continually break any automation.
Integration. Integrating your automation and development teams will save your project countless hours of ineffective churn. Open the lines of communication early, so they can directly address any inefficiencies. Take for example web element IDs. If your developers aren’t adding these critical elements to their code, there will be nothing for your QA team to call in their test scripts. Digging deeper, there is an additional way to access the web elements, but it’s more difficult and also slows down the tests. Connecting the quality and development perspectives will eliminate wasted time allowing the team to focus on higher-value work.
Software development |
---|
Core activities |
Paradigms and models |
Methodologies and frameworks |
Supporting disciplines |
Practices |
Tools |
Standards and Bodies of Knowledge |
Glossaries |
In software testing, test automation is the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes.[1] Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually. Test automation is critical for continuous delivery and continuous testing.
There are many approaches to test automation, however below are the general approaches used widely:
- Graphical user interface testing. A testing framework that generates user interface events such as keystrokes and mouse clicks, and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct.
- API driven testing. A testing framework that uses a programming interface to the application to validate the behaviour under test. Typically API driven testing bypasses application user interface altogether. It can also be testing public (usually) interfaces to classes, modules or libraries are tested with a variety of input arguments to validate that the results that are returned are correct.
One way to generate test cases automatically is model-based testing through use of a model of the system for test case generation, but research continues into a variety of alternative methodologies for doing so.[citation needed] In some cases, the model-based approach enables non-technical users to create automated business test cases in plain English so that no programming of any kind is needed in order to configure them for multiple operating systems, browsers, and smart devices.[2]
What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team must make.[3] A multi-vocal literature review of 52 practitioner and 26 academic sources found that five main factors to consider in test automation decision are: 1) System Under Test (SUT), 2) the types and numbers of tests, 3) test-tool, 4) human and organizational topics, and 5) cross-cutting factors. The most frequent individual factors identified in the study were: need for regression testing, economic factors, and maturity of SUT.[4]
A growing trend in software development is the use of unit testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) that allow the execution of unit tests to determine whether various sections of the code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected.
Test automation, mostly using unit testing, is a key feature of extreme programming and agile software development, where it is known as test-driven development (TDD) or test-first development. Unit tests can be written to define the functionality before the code is written. However, these unit tests evolve and are extended as coding progresses, issues are discovered and the code is subjected to refactoring.[5] Only when all the tests for all the demanded features pass is the code considered complete. Proponents argue that it produces software that is both more reliable and less costly than code that is tested by manual exploration.[citation needed] It is considered more reliable because the code coverage is better, and because it is run constantly during development rather than once at the end of a waterfall development cycle. The developer discovers defects immediately upon making a change, when it is least expensive to fix. Finally, code refactoring is safer when unit testing is used; transforming the code into a simpler form with less code duplication, but equivalent behavior, is much less likely to introduce new defects when the refactored code is covered by unit tests.
Some software testing tasks (such as extensive low-level interface regression testing) can be laborious and time-consuming to do manually. In addition, a manual approach might not always be effective in finding certain classes of defects. Test automation offers a possibility to perform these types of testing effectively.
Once automated tests have been developed, they can be run quickly and repeatedly. Many times, this can be a cost-effective method for regression testing of software products that have a long maintenance life. Even minor patches over the lifetime of the application can cause existing features to break which were working at an earlier point in time.
Test automation tools can be expensive and are usually employed in combination with manual testing. Test automation can be made cost-effective in the long term, especially when used repeatedly in regression testing. A good candidate for test automation is a test case for common flow of an application, as it is required to be executed (regression testing) every time an enhancement is made in the application. Test automation reduces the effort associated with manual testing. Manual effort is needed to develop and maintain automated checks, as well as reviewing test results.
In automated testing, the test engineer or software quality assurance person must have software coding ability since the test cases are written in the form of source code which when run produce output according to the assertions that are a part of it. Some test automation tools allow for test authoring to be done by keywords instead of coding, which do not require programming.
- 4Framework approach in automation
API driven testing[edit]
API testing is also being widely used by software testers due to the difficulty of creating and maintaining GUI-based automation testing. It involves directly testing APIs as part of integration testing, to determine if they meet expectations for functionality, reliability, performance, and security.[6] Since APIs lack a GUI, API testing is performed at the message layer.[7] API testing is considered critical when an API serves as the primary interface to application logic since GUI tests can be difficult to maintain with the short release cycles and frequent changes commonly used with agile software development and DevOps.[8][9]
Continuous testing[edit]
Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.[10][11] For Continuous Testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.[12]
Graphical User Interface (GUI) testing[edit]
Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. The advantage of this approach is that it requires little or no software development. This approach can be applied to any application that has a graphical user interface. However, reliance on these features poses major reliability and maintainability problems. Relabelling a button or moving it to another part of the window may require the test to be re-recorded. Record and playback also often adds irrelevant activities or incorrectly records some activities.[citation needed]
A variation on this type of tool is for testing of web sites. Here, the 'interface' is the web page. However, such a framework utilizes entirely different techniques because it is rendering HTML and listening to DOM Events instead of operating system events. Headless browsers or solutions based on Selenium Web Driver are normally used for this purpose.[13][14][15]
Another variation of this type of test automation tool is for testing mobile applications. This is very useful given the number of different sizes, resolutions, and operating systems used on mobile phones. For this variation, a framework is used in order to instantiate actions on the mobile device and to gather results of the actions.[16][better source needed]
Another variation is script-less test automation that does not use record and playback, but instead builds a model[clarification needed] of the application and then enables the tester to create test cases by simply inserting test parameters and conditions, which requires no scripting skills.
Framework approach in automation[edit]
A test automation framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort.
The main advantage of a framework of assumptions, concepts and tools that provide support for automated software testing is the low cost for maintenance. If there is change to any test case then only the test case file needs to be updated and the driver Script and startup script will remain the same. Ideally, there is no need to update the scripts in case of changes to the application.
Choosing the right framework/scripting technique helps in maintaining lower costs. The costs associated with test scripting are due to development and maintenance efforts. The approach of scripting used during test automation has effect on costs.
Various framework/scripting techniques are generally used:
- Linear (procedural code, possibly generated by tools like those that use record and playback)
- Structured (uses control structures - typically ‘if-else’, ‘switch’, ‘for’, ‘while’ conditions/ statements)
- Data-driven (data is persisted outside of tests in a database, spreadsheet, or other mechanism)
- Hybrid (two or more of the patterns above are used)
- Agile automation framework
The Testing framework is responsible for:[17]
- defining the format in which to express expectations
- creating a mechanism to hook into or drive the application under test
- executing the tests
- reporting results
Test automation interface[edit]
Test automation interface are platforms that provide a single workspace for incorporating multiple testing tools and frameworks for System/Integration testing of application under test. The goal of Test Automation Interface is to simplify the process of mapping tests to business criteria without coding coming in the way of the process. Test automation interface are expected to improve the efficiency and flexibility of maintaining test scripts.[18]
Test Automation Interface consists of the following core modules:
- Interface Engine
- Interface Environment
- Object Repository
Interface engine[edit]
Interface engines are built on top of Interface Environment. Interface engine consists of a parser and a test runner. The parser is present to parse the object files coming from the object repository into the test specific scripting language. The test runner executes the test scripts using a test harness.[18]
Object repository[edit]
Object repositories are a collection of UI/Application object data recorded by the testing tool while exploring the application under test.[18]
Defining boundaries between automation framework and a testing tool[edit]
Tools are specifically designed to target some particular test environment, such as Windows and web automation tools, etc. Tools serve as a driving agent for an automation process. However, an automation framework is not a tool to perform a specific task, but rather infrastructure that provides the solution where different tools can do their job in a unified manner. This provides a common platform for the automation engineer.
There are various types of frameworks. They are categorized on the basis of the automation component they leverage. These are:
- Code-driven testing
What to test[edit]
Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with test oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
One must keep satisfying popular requirements when thinking of test automation:
- Platform and OS independence
- Data driven capability (Input Data, Output Data, Metadata)
- Customization Reporting (DB Data Base Access, Crystal Reports)
- Easy debugging and logging
- Version control friendly – minimal binary files
- Extensible & Customization (Open APIs to be able to integrate with other tools)
- Common Driver (For example, in the Java development ecosystem, that means Ant or Maven and the popular IDEs). This enables tests to integrate with the developers' workflows.
- Support unattended test runs for integration with build processes and batch runs. Continuous integration servers require this.
- Email Notifications like bounce messages
- Support distributed execution environment (distributed test bed)
- Distributed application support (distributed SUT)
See also[edit]
References[edit]
- ^Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. p. 74. ISBN978-0-470-04212-0.
- ^Proceedings from the 5th International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. 'Test Design: Lessons Learned and Practical Implications. doi:10.1109/IEEESTD.2008.4578383. ISBN978-0-7381-5746-7.
- ^Brian Marick. 'When Should a Test Be Automated?'. StickyMinds.com. Retrieved 2009-08-20.
- ^Garousi, Vahid; Mäntylä, Mika V. (2016-08-01). 'When and what to automate in software testing? A multi-vocal literature review'. Information and Software Technology. 76: 92–117. doi:10.1016/j.infsof.2016.04.015.
- ^Learning Test-Driven Development by Counting Lines; Bas Vodde & Lasse Koskela; IEEE Software Vol. 24, Issue 3, 2007
- ^Testing APIs protects applications and reputations, by Amy Reichert, SearchSoftwareQuality March 2015
- ^All About API Testing: An Interview with Jonathan Cooper, by Cameron Philipp-Edmonds, Stickyminds August 19, 2014
- ^The Forrester Wave™ Evaluation Of Functional Test Automation (FTA) Is Out And It's All About Going Beyond GUI Testing, by Diego Lo Giudice, Forrester April 23, 2015
- ^Produce Better Software by Using a Layered Testing Strategy, by Sean Kenefick, Gartner January 7, 2014
- ^Part of the Pipeline: Why Continuous Testing Is Essential, by Adam Auerbach, TechWell Insights August 2015
- ^The Relationship between Risk and Continuous Testing: An Interview with Wayne Ariola, by Cameron Philipp-Edmonds, Stickyminds December 2015
- ^DevOps: Are You Pushing Bugs to Clients Faster, by Wayne Ariola and Cynthia Dunlop, PNSQC October 2015
- ^Headless Testing with Browsers; https://docs.travis-ci.com/user/gui-and-headless-browsers/
- ^Headless Testing with PhantomJS;http://phantomjs.org/headless-testing.html
- ^Automated User Interface Testing; https://www.devbridge.com/articles/automated-user-interface-testing/
- ^Testmunk. 'A Beginner's Guide to Automated Mobile App Testing | Testmunk Blog'. blog.testmunk.com. Retrieved 2015-09-17.
- ^'Selenium Meet-Up 4/20/2010 Elisabeth Hendrickson on Robot Framework 1of2'. Retrieved 2010-09-26.
- ^ abc'Conquest: Interface for Test Automation Design'(PDF). Retrieved 2011-12-11.
- Notes
- Elfriede Dustin; et al. (1999). Automated Software Testing. Addison Wesley. ISBN978-0-201-43287-9.
- Elfriede Dustin; et al. (2009). Implementing Automated Software Testing. Addison Wesley. ISBN978-0-321-58051-1.
- Mark Fewster & Dorothy Graham (1999). Software Test Automation. ACM Press/Addison-Wesley. ISBN978-0-201-33140-0.
- Roman Savenkov: How to Become a Software Tester. Roman Savenkov Consulting, 2008, ISBN978-0-615-23372-7
- Hong Zhu; et al. (2008). AST '08: Proceedings of the 3rd International Workshop on Automation of Software Test. ACM Press. ISBN978-1-60558-030-2.
- Mosley, Daniel J.; Posey, Bruce (2002). Just Enough Software Test Automation. ISBN978-0130084682.
- Hayes, Linda G., 'Automated Testing Handbook', Software Testing Institute, 2nd Edition, March 2004
- Kaner, Cem, 'Architectures of Test Automation', August 2000
External links[edit]
- Test Automation Snake Oil by James Bach
- When Should a Test Be Automated? by Brian Marick
- Success Factors for Keyword Driven Testing by Hans Buwalda
- Automation That Learns: Making Your Computer Work for You by Jeremy Carey-Dressler
- Automation Testing Resources & Best Practices by Joe Colantonio
Choosing the right set of automation tools is paramount in successful test automation. Find an overview of various tools strengths and weaknesses here.
Join the DZone community and get the full member experience.
Join For FreeA defining factor for successfully applying test automation in software projects is choosing and using the right set of test automation tools. This is a daunting task, especially for those new to software test automation, because there are so many tools in the market to choose from, each having different strengths and weaknesses. There is no one tool that can fit all automated testing needs, which makes finding the right tool difficult. Learn how to identify the right automation tool for your project with this qualitative comparison popular automated testing toolsets in the market.
Overview of Tools
Katalon Studio is an automated testing platform that offers a comprehensive set of features to implement full automated testing solutions for mobile and web applications. Built on top of the open-source Selenium and Appium frameworks, Katalon allows teams to get started with test automation quickly by reducing the effort and expertise required for learning and integrating these frameworks for automated testing needs.
Selenium is the most popular automation framework that consists of many tools and plugins for Web application testing. Selenium is known for its powerful capability to support performance testing of Web applications. Selenium is the most popular choice in the open-source test automation space, partly due to its large and active development and user community.
HP Unified Functional Testing (UFT), formerly QuickTest Professional (QTP), is probably the most popular commercial tool for functional test automation. HP UFT offers a comprehensive set of features that can cover most functional automated testing needs on the desktop, mobile, and web platforms.
TestComplete is also a commercial integrated platform for desktop, mobile, and web application testing. Like UFT, TestComplete offers a number of key test automation features such as keyword-driven and>Comparison of Tools
The table below provides a comparison of the tools based on the key features of software automation:
Features | Katalon Studio | Selenium | UFT (QTP) | TestComplete |
Test development platform | Cross-platform | Cross-platform | Windows | Windows |
Application under test | Web and mobile apps | Web apps | Windows desktop, web, mobile apps | Windows desktop, web, mobile apps |
Scripting languages | Java/Groovy | Java, C#, Perl, Python, JavaScript, Ruby, PHP | VBScript | JavaScript, Python, VBScript, JScript, Delphi, C++, and C# |
Programming skills | Not required. Recommended for advanced test scripts | Advanced skills needed to integrate various tools | Not required. Recommended for advanced test scripts | Not required. Recommended for advanced test scripts |
Learning curves | Medium | High | Medium | Medium |
Ease of installation and use | Easy to set up and run | Require installing and integrating various tools | Easy to setup and run | Easy to setup and run |
Script creation time | Quick | Slow | Quick | Quick |
Object storage and maintenance | Built-in object repository, XPath, object re-identification | XPath, UI Maps | Built-in object repository, smart object detection and correction | Built-in object repository, detecting common objects |
Image-based testing | Built-in support | Require installing additional libraries | Built-in support, image-based object recognition | Built-in support |
Continuous integrations | Popular CI tools (e.g. Jenkins, Teamcity) | Various CI tools (e.g. Jenkins, Cruise Control) | Various CI tools (e.g. Jenkins, HP Quality Center) | Various CI tools (e.g. Jenkins, HP Quality Center) |
Product support | Ticketing support, community | Open source community | Dedicated staff, community | Dedicated staff, community |
License type | Freeware | Open source (Apache 2.0) | Proprietary | Proprietary |
Cost | Free | Free | License and maintenance fees | License and maintenance fees |
Strengths and Weaknesses
Below is a summary of key strengths and limitations of the tools, based on the comparison above.
Tools | Strengths | Limitations |
Katalon Studio | No licensing and maintenance fees requires Integrating necessary frameworks and features for quick test cases creation and execution. Built on top of the Selenium framework but eliminating the need for advanced programming skills required for Selenium. | Emerging solution with a small community Feature set is still evolving. Lack of choices for scripting languages: only Java/Groovy is supported. |
Selenium | Open source, no licensing and maintenance fees Large and active development and user community to keep pace with software technologies. Open for integration with other tools and frameworks to enhance its capability | Testing teams need to have good programming skills and experience to setup and integrate Selenium with other tools and frameworks. New teams need to invest time upfront for setup and integration Slow support from the community. |
UFT | Mature, comprehensive automated testing features integrated into a single system. Dedicated user support plus an established large user community. Requiring only basic programming skills to get started with test creation and execution. | Costly solution: license and maintenance fees are considerably high. Possible high costs for upgrades and additional modules. Supporting only VBScript. |
TestComplete | Mature, comprehensive automated testing features integrated into a single system. Many scripting languages to choose from. Only basic programming skills needed. | Like UFT, considerable licensing and maintenance fees needed for TestComplete. |
There is no one-size-fits-all tool for automated testing. It is highly recommended that testers evaluate various tools in order to select what would best meet their automated testing needs. Programming languages and technologies used to develop software continue to evolve, as do the automated testing tools, making cost a significant factor in tool selection. Commercial vendors often charge for tool upgrades, which can be substantial if your software uses emerging and frequently changing technologies. Open source and non-commercial tools, on the other hand, do not incur additional charges, but require effort and expertise for integrating new upgrades. It is difficult to find the support and expertise needed for integrating various tools and frameworks into open-source solutions. Emerging tools that integrate with open-source frameworks, like Katalon, offer a viable alternative to both commercial and open-source automated testing solutions.
Like This Article? Read More From DZone
Published at DZone with permission of Vu Nguyen . See the original article here.
Opinions expressed by DZone contributors are their own.
InfoQ HomepageNewsBDD Tool Cucumber is Not a Testing Tool
Leia em português
This item in chinese
If you are using Cucumber just for automated testing you can do better. By writing scenarios that illustrate business rules instead of UI functionality you can get business analysts involved and writing scenarios before code enables programmers to be guided by an unambiguous specification. Then you are doing Behaviour-Driven Development (BDD) Aslak Hellesøy states reflecting over his experience of seeing it misused and misunderstood.
Hellesøy, who created Cucumber in 2008 with 5 million downloads during the first three years, emphasizes that Cucumber first and foremost is a collaboration tool aiming for a common understanding across all members of a team. Cucumber features should be written before the code implementing the feature. When you work with BDD writing examples, regression tests are a by-product, the testing is not the activity.
Looking at Cucumber for JavaScript, Julien Biezemans sees benefits in BDD coming to web development but notes that the same misunderstandings Hellesøy mentions leading to using Cucumber only as a testing tool are common here also. Also for Biezemans, BDD is about encouraging conversation between all involved, writing down examples to make things clearer and reduce ambiguity allowing everyone to agree on what they are building. Automating the scenarios resulting from the conversations is an optional next step.
Hellesøy declared last year that to use Cucumber you have to follow a process involving most people in a software team. That process is BDD which later was renamed by Gojko Adzic to Specification by Example. Somewhat simplified this process consists of two main activities:
- Specification Workshops where business analysts responsible for the requirements together with programmers and testers discuss features to be developed, (these three roles are often called the three amigos). At the same time they write down examples of how the software should behave, as Cucumber scenarios.
- Outside-In Development where programmers incrementally write code and run the scenarios using Cucumber until the feature passes all the tests. Programmers typically start with functionality closest to the user working their way inwards to core domain, hence the name of the activity.
Liz Keogh notes that defining BDDis hard; with a methodology derived from many other methods and philosophies she finds it difficult to draw a boundary on what is respectively is not included. Instead Keogh thinks of BDD as an anchor term, at its core are conversation, collaboration and scenarios together with ways to automate them. Around the core are a number of other practices like those mentioned by Hellesøy and many tools including Cucumber, JBehave and SpecFlow. Defining BDD Keogh therefore uses a phrase describing the anchor idea:
Using examples in conversation to illustrate behaviour
Related Topics:
Related Editorial
From Docker to Kubernetes: Container Networking 101 (By O’Reilly)
Related Vendor Content
Related Sponsor
NGINX Plus is the complete application delivery platform for the modern web. Start your 30 day free trial.
The InfoQ Newsletter
A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example