What is COTS?
COTS stands for “Commercial Off-The-Shelf” and is often used in reference to software products or tools supplied by third-party vendors. These COTS test automation tools are created to help you in automating some or all of your testing, or to increase your ability to perform specific types of testing.
Another option for automation is to consider scripting languages. Scripting languages offer powerful tool kits that can be used to accomplish many of the testing tasks offered by many off-the-shelf tools.
Regardless of which you use, approach your automation effort as a serious development project and the advantages can be great and the disadvantages largely mitigated.
Below we provide you with some analysis of the advantages and disadvantages of both.
Advantages of COTS Test Automation Tools
- Shorter Initial Implementation Cycle: many COTS test automation tools are quick to install, configure and use, and thus COTS generated tests that can be more quickly added to your testing cycle than individually created scripts.
- Availability of Specific and Customized Training: training, provided by the vendor, is generally available although sometimes at a prohibitively high cost.
- Integration with Other Tools: many COTS tool vendors make many different tools, not just test automation tools. If you select a COTS test automation tool from a leading tool vendor, it is very likely that they also offer additional tools such as test case management tools, defect tracking tools, source control tools and even requirements management tools. These “suites” of tools can, when used effectively, streamline the flow of information between these tools and the people that use them.
- Mature Product Functionality: most major COTS test automation tools have been evolving along with the industry over the course of many years and will contain many of the features and functionality automators are looking for.
Disadvantages of COTS Test Automation Tools
- Cost: Purchasing and licensing COTS test automation tools can be quite expensive. You need to consider also the cost of the support contract, which is can be between 15 to 30% of the cost of the tool license per year.
- Complexity: many COTS test automation tools provide a large amount of functionality and support a wide variety of technologies and environments, but in reality you end up using very little of the functionality and do not need all environments supported. Accessing this advanced functionality can require significant programming skills in some cases.
- Value (functionality as compared to cost): most organizations need to see a return on investment (ROI) – the money and time spent on automation using a COTS test tool. Sometimes, the ROI for a COTS test automation tool can be so lengthy (often over 2 years), that it is simply not worth an organization’s effort to implement such a tool choice.
- Slow adoption of new technologies: with each new technological advance, the tools must be extended to handle these new technologies. This can often be six months to two years later than when the new technology first appeared on your horizon. In the meantime, you will have only a partial solution or perhaps one that has serious defects or deficiencies.
- Proprietary scripting languages and data formats: most COTS test automation tools store data such as test results and script details in a proprietary format. Some of these formats are usually “flat” but even so it can be a problem if you want to change tools and you do not want to have to copy and paste all of your scripts. Test results and other related data could also be difficult to extract. Most COTS test tools will have some reporting functionality built-in, but this may not be what you want or need in all cases.
- Environment and Configuration Problems are Common: it is a common occurrence to find that scripts you created, that run just fine on your environment, will fail on your fellow tester’s environment. Many COTS testing tools rely heavily on specific environment information, such as operating system, screen size/resolution, color settings, locations of network connections or installed programs. Some tools will hard code this specific information into your test scripts, while others incorporate this information into the tool’s operating environment (“settings”).
Advantages of Scripting Languages
- Cost Savings: A Perl license costs zero dollars. A Java license costs zero dollars. When you compare the cost of licensing a tool (money) and learning how to use it (time) versus learning how to script effectively (time and maybe some money for books or courses), you can spend less on the scripting alternative.
- Flexibility: scripting languages allows automation to grow as your product grows supplying the needed test-oriented functionality. The functionality of your automation is entirely up to you. Whatever testing you want to do, and however you want to do that automation, you have virtually unlimited possibilities. Also scripts are easy to learn, expand and enhance when written in languages such as Perl or Visual Basic (VB), for example.
- Portability: scripting languages can be easily used in different operating systems; scripts are portable between versions of scripting languages.
- Non-proprietary formats for data/test scripts: you can determine what format data is stored, and how the test parameters are stored, read and written. The data is yours; you can control the formatting, the storage, and the retrieval of it. You can also control any real-time data reduction or analysis.
- Easier integration with new(er) technologies: face=Verdana>while tool vendors have to “catch up” to new technologies, scripting languages, like programming languages, are at the front of the technology curve. They are the first things to change (followed by OS’s, then applications to run on the OS’s) and to incorporate new technologies.
- Many more resources available: there are lots of books, on-line books, forums and user groups, etc. available. Plus you can get some great help from your internal developers. They are usually more than happy to help you out with a scripting or programming problem. In some cases, it can be easier to get assistance from development with a specific aspect of your automation if you are using scripting, as opposed to a COTS tool with which most developers are unfamiliar (and rarely have the time to learn).
Disadvantages of Scripting Languages
- Steep learning curve: although we feel that most scripting languages are not that difficult to learn, this depends heavily on the person learning to script. If they have any prior scripting or programming experience, the time spent learning the new scripting language is far less than that of someone who has never done any scripting or programming. Also there are people who find this kind of abstract thinking to be difficult, new or just plain annoying. Many people feel that it simply takes too long to accomplish what they can by using a COTS tool.
- Methodology: to take full advantage of using scripting languages for testing, and for those scripts to be effective, you need a deep understanding of both test methodology and software engineering principles. Not every organization has such a dually-skilled resource.
- Functionality: when you create your own scripts, you do not have the help from wizards or report templates. Though you can leverage existing libraries, you will have to create your functionality from scratch.
- Testing: just like any other piece of software, you need to test your scripts. You need to verify that the scripts are performing as expected, and that the results or other data that the scripts generate are meaningful. You need time to prepare data and testing. You may need other resources to help you with testing so you can get a different perspective.
Summary
There is no true “silver bullet” to your test automation needs. Approach any COTS test automation tools selection process with your eyes wide open and be aware of the wide range of pluses and minuses you must consider in making your choice.
For related reading, check out these articles:
