COTS Tools vs. Scripting Languages for Test Automation

What is COTS?

COTS stands for “Commercial Off-The-Shelf” and is often used in reference to software products or tools supplied by third-party vendors. These COTS test automation tools are created to help you in automating some or all of your testing, or to increase your ability to perform specific types of testing.

Another option for automation is to consider scripting languages. Scripting languages offer powerful tool kits that can be used to accomplish many of the testing tasks offered by many off-the-shelf tools.


Regardless of which you use, approach your automation effort as a serious development project and the advantages can be great and the disadvantages largely mitigated.

Below we provide you with some analysis of the advantages and disadvantages of both.

Advantages of COTS Test Automation Tools

  • Shorter Initial Implementation Cycle: many COTS test automation tools are quick to install, configure and use, and thus COTS generated tests that can be more quickly added to your testing cycle than individually created scripts.
  • Availability of Specific and Customized Training: training, provided by the vendor, is generally available although sometimes at a prohibitively high cost.
  • Integration with Other Tools: many COTS tool vendors make many different tools, not just test automation tools. If you select a COTS test automation tool from a leading tool vendor, it is very likely that they also offer additional tools such as test case management tools, defect tracking tools, source control tools and even requirements management tools. These “suites” of tools can, when used effectively, streamline the flow of information between these tools and the people that use them.
  • Mature Product Functionality: most major COTS test automation tools have been evolving along with the industry over the course of many years and will contain many of the features and functionality automators are looking for.

Disadvantages of COTS Test Automation Tools

  • Cost: Purchasing and licensing COTS test automation tools can be quite expensive. You need to consider also the cost of the support contract, which is can be between 15 to 30% of the cost of the tool license per year.
  • Complexity: many COTS test automation tools provide a large amount of functionality and support a wide variety of technologies and environments, but in reality you end up using very little of the functionality and do not need all environments supported. Accessing this advanced functionality can require significant programming skills in some cases.
  • Value (functionality as compared to cost): most organizations need to see a return on investment (ROI) – the money and time spent on automation using a COTS test tool. Sometimes, the ROI for a COTS test automation tool can be so lengthy (often over 2 years), that it is simply not worth an organization’s effort to implement such a tool choice.
  • Slow adoption of new technologies: with each new technological advance, the tools must be extended to handle these new technologies. This can often be six months to two years later than when the new technology first appeared on your horizon. In the meantime, you will have only a partial solution or perhaps one that has serious defects or deficiencies.
  • Proprietary scripting languages and data formats: most COTS test automation tools store data such as test results and script details in a proprietary format. Some of these formats are usually “flat” but even so it can be a problem if you want to change tools and you do not want to have to copy and paste all of your scripts. Test results and other related data could also be difficult to extract. Most COTS test tools will have some reporting functionality built-in, but this may not be what you want or need in all cases.
  • Environment and Configuration Problems are Common: it is a common occurrence to find that scripts you created, that run just fine on your environment, will fail on your fellow tester’s environment. Many COTS testing tools rely heavily on specific environment information, such as operating system, screen size/resolution, color settings, locations of network connections or installed programs. Some tools will hard code this specific information into your test scripts, while others incorporate this information into the tool’s operating environment (“settings”).

Advantages of Scripting Languages

  • Cost Savings: A Perl license costs zero dollars. A Java license costs zero dollars. When you compare the cost of licensing a tool (money) and learning how to use it (time) versus learning how to script effectively (time and maybe some money for books or courses), you can spend less on the scripting alternative.
  • Flexibility: scripting languages allows automation to grow as your product grows supplying the needed test-oriented functionality. The functionality of your automation is entirely up to you. Whatever testing you want to do, and however you want to do that automation, you have virtually unlimited possibilities. Also scripts are easy to learn, expand and enhance when written in languages such as Perl or Visual Basic (VB), for example.
  • Portability: scripting languages can be easily used in different operating systems; scripts are portable between versions of scripting languages.
  • Non-proprietary formats for data/test scripts: you can determine what format data is stored, and how the test parameters are stored, read and written. The data is yours; you can control the formatting, the storage, and the retrieval of it.  You can also control any real-time data reduction or analysis.
  • Easier integration with new(er) technologies: face=Verdana>while tool vendors have to “catch up” to new technologies, scripting languages, like programming languages, are at the front of the technology curve. They are the first things to change (followed by OS’s, then applications to run on the OS’s) and to incorporate new technologies.
  • Many more resources available: there are lots of books, on-line books, forums and user groups, etc. available. Plus you can get some great help from your internal developers. They are usually more than happy to help you out with a scripting or programming problem. In some cases, it can be easier to get assistance from development with a specific aspect of your automation if you are using scripting, as opposed to a COTS tool with which most developers are unfamiliar (and rarely have the time to learn).

Disadvantages of Scripting Languages

  • Steep learning curve: although we feel that most scripting languages are not that difficult to learn, this depends heavily on the person learning to script. If they have any prior scripting or programming experience, the time spent learning the new scripting language is far less than that of someone who has never done any scripting or programming. Also there are people who find this kind of abstract thinking to be difficult, new or just plain annoying. Many people feel that it simply takes too long to accomplish what they can by using a COTS tool.
  • Methodology: to take full advantage of using scripting languages for testing, and for those scripts to be effective, you need a deep understanding of both test methodology and software engineering principles. Not every organization has such a dually-skilled resource.
  • Functionality: when you create your own scripts, you do not have the help from wizards or report templates. Though you can leverage existing libraries, you will have to create your functionality from scratch.
  • Testing: just like any other piece of software, you need to test your scripts. You need to verify that the scripts are performing as expected, and that the results or other data that the scripts generate are meaningful. You need time to prepare data and testing. You may need other resources to help you with testing so you can get a different perspective.

Summary

There is no true “silver bullet” to your test automation needs. Approach any COTS test automation tools selection process with your eyes wide open and be aware of the wide range of pluses and minuses you must consider in making your choice.

For related reading, check out these articles:

Posted in  All, Automation & Tools, Planning for Quality | Tagged , , | Comments Off on COTS Tools vs. Scripting Languages for Test Automation

Risk-based Testing – Targeting the Risk

One of the obvious truths about software testing is that there will never be enough time to do all the tests.  Enter risk-based testing.

While everything should be tested, limited resources and schedule require that testing is structured whereby the most critical areas of the software are tested first and most thoroughly.

Performing risk-based analysis on the software will allow intelligent choices to be made regarding what tests need to be run in a time crunch situation. It ensures that the choice between what is tested and what is not is a conscious decision based on an understanding of the technology and the users.

Taking the understanding of “what is risk?“, answer the following questions with as many relevant points as possible:

  • In what areas is the user most likely to experience a problem?
  • What would be the impact of a certain type of problem?
  • What is the relative testing priority of each type of potential problem?

Get the ideas of the end-user, the application architects, the developers, customer support – ask lots of questions and dig up internal historical data or industry metrics.

Once the risk areas, impact and priorities have been defined, a strategy can be conceived for how to approach the overall test effort that will maximize the value of your testing budget.

Having this test strategy in advance of actual test execution means that what to test and how long it is estimated to take is known: resources can be obtained, test environments can be set-up, test cases detailed, and test harnesses constructed.

Plan to concentrate most of your testing and attention on areas with known complexity, usage patterns, and technical risk. The accuracy, reliability and readiness of the application for release can be measured much more accurately than just through overall bug counts.

Look at the schedule and determine how much time there really is to spend on testing. Identify and prioritize the risk areas of the application.  Create a test strategy that maximizes the resources and time available.  Remember that risks can change throughout a project and so the test strategy must change to deal with them. Don’t get locked into the original test strategy as the _only_ strategy that will work for the project.

In the case of shifting milestones, the strategy can be easily modified to suit the time left and the level of risk that is acceptable when _planning_ not to execute certain tests.

The testing efforts are now focused on portions of the software where the risk of potential failure is greatest, or where potential failure would have the greatest impact. The result of this testing approach provides everyone with confidence in how well the software will hold up in the hands of the end-user.

Summary

With the increasing complexity of modern software and the shrinking time line for development and testing, it is important to get the most out of your testing efforts. With the understanding that it’s impossible to test an application and find every bug, the technique of risk-based testing provides the coverage necessary to find the most important (and most costly if not found and fixed) bugs with the least effort.

Posted in  All, Planning for Quality, Risk & Testing, Test Planning & Strategy | Tagged , , , | Comments Off on Risk-based Testing – Targeting the Risk

Components of a Test Strategy

A Test Strategy is a documented approach to testing where the test effort, test domain, test configurations, and test tools employed to verify and validate a set of functionality are defined. It also includes information on schedules, resource allocations, and staff utilization. This information is crucial to allow the test team (Test) to be as organized and effective as possible.

A Test Strategy is not the same as a Test Plan, which is a document that collects and organizes test cases by functional areas and/or types of testing in a form that can be presented to the other teams and/or customer.

Both are important pieces of the Quality Assurance process since they help communicate the test approach scope and ensure test coverage while improving the efficiency of the testing effort.


What is in the Test Strategy?

The following is a list of some of the sections that are typically included in the Test Strategy document.

  • Introduction – contains an overview of the project, lists related documents and references, document conventions, and assumptions made in the course of creating the strategy.
  • Scope – describes the scope of the test team’s involvement in the project; describes the test areas Test is responsible for and why. It also defines the areas for which Test is not responsible.
  • Resources & Schedule for Test Activities – describes the resources available and their roles. Includes a schedule overview for the project, making sure the estimated time for the testing activities and milestone dates are present. The build schedule can also be included if available.
  • Acceptance Criteria – defines the minimum criteria that a product must achieve before it is shipped.
  • Test environment – describes the hardware and software platforms that are used for testing, including Client/Server configuration, Network, etc…and what will be tested in each platform.
  • Tools – describes the tools used for test case management, defect reporting and test automation.
  • Test Priorities – describes the priorities of the test effort during the test planning, test automation, test data creation, and test execution phases.
  • Test Planning – describes such activities as requirements review and test analysis to determine a list of suitable tests required for verification of the product. It also describes how the tests are expanded into full test cases, complete with descriptions, reproduction steps, and expected results.
  • Executing a Test Pass – describes how the test pass execution is performed, and when the testing is executed, in accordance with the types of testing to be performed. For example, test cases that are critical are tested first to ensure the build has the minimum functionality required before further testing.
  • Types of testing to be performed – defines the different types of testing to be performed, and the extent to which Test will be carrying out each type of testing. The most common types of testing types are:
    • Build Verification Tests
    • Functionality Testing
    • User Interface Testing
    • Usability Testing
    • Error Handling
    • System Platform
    • Stress Testing
    • Performance Testing
    • Installation Testing
    • Print Testing
    • Localization Testing
    • Regression testing
  • Risks and Issues – lists of outstanding risks and issues related to the test effort.

Benefiting from the Test Strategy

The main groups that benefit from having a Test Strategy are the testing team, development and project management, but other groups such as user ed and marketing can also benefit from the information contained in the Test Strategy.

  • Test Team: The Test team will follow the Test Strategy and make sure testing is performed in accordance with the plan. They will also analyze the results and make recommendations on the quality of the functionality. The Test Strategy document should help the Test Team answer the questions below:
    • Do I have all documentation I need to start the test planning?
    • Is the time scheduled for testing planning adequate?
    • Do I have the tools to develop the test cases? To log defects?
    • Who is going to review the test analysis/ test planning and when?
    • Do I have all I need to start testing (equipment/tools)?
    • Do I have all the data/files I need to start testing?
    • Do I know the functionality I will test on each build?
    • Is all functionality being covered during all phases?
    • What are the procedures if I find a serious defect?
  • Development: Development will understand the functionality being tested and the conditions under which these tests are to be performed. The questions that the Test Strategy document should answer are:
    • What is the overall approach to testing on this project?
    • Who is responsible for the different types of testing, particularly Unit and Integration testing?
    • Do I have time scheduled for reviewing the test plans for depth of coverage?
    • Is the test environment adequate relative to the intended production environment?
  • Project Management: Project Management will understand the information regarding configurations (hardware and software) under which the product was verified and validated, and the procedure for assessing the quality of the product based on the type of testing being performed. They are also informed about the testing schedule and its impact on the deadlines. The Test Strategy should help with the following questions:
    • Do I need to hire more people during the test planning or testing phase?
    • Do we have all the hardware and software required for testing?
    • Do we have the tools required for test planning and defect reporting?
    • If a new tool is required, is the time needed for training the testing team scheduled?
    • Are all types of testing defined as required?
    • Are all the testing tasks well defined?
    • Are the testing priorities clear for each phase of the project?
    • Are there enough test execution passes for each phase of the project?
    • What are the issues and risks identified by Test that are still outstanding?

There is another important document whose purpose is very often confused with the Test Strategy or Test Plan; and that is the QA Plan. The QA Plan is intended to be a high level document that clearly outlines the boundaries of QA’s responsibilities on the project relative to the rest of the project personnel, including any clients, sub-contractors, and co-contractors.

The QA Plan includes descriptions of methodologies, practices, standards, quality requirements, metrics, reviews and audits, code control, media control, etc., in addition to outlining the basics of the responsibilities of the Test Team.

The Test Strategy draws upon this parent document and its information, if available, and further details the responsibilities of the Test Team and its approach to testing.

For related reading, check out these articles:

Posted in  All, Test Planning & Strategy | Tagged , | Comments Off on Components of a Test Strategy

Test Throughout the Development Lifecycle

There are many perceptions of what software testing is all about. Some people see software testing as something developers do as they write code. For some testing is performed all at once when the code is complete. Others believe that testing should be performed along-side development activities.

It is a well-established fact that the sooner a defect is found the less expensive it is to fix. Consider the definition: “Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results” – The Complete Guide to Software Testing, Bill Hetzel, 1988. Testing is a set of continuous activities performed with the objective of evaluating the software against pre-defined results/expectations.

Testing the application throughout its development lifecycle, emphasizing the early lifecycle activities helps increase the overall effectiveness and efficiency of this effort. Further, early involvement allows evaluation of important planning, design and development decisions with respect to how these decisions aid or impair the testability of the application.

Stages of the Software Development

Test resources have at least one associated activity within any phase of a project lifecycle regardless of the development methodology followed:

Project Planning – When the project plan and associated schedule is created, Testing reviews the plan to ensure that all testing tasks have been included and have been scheduled for an appropriate length of time. As well, assigning resources to these tasks will allow any potential resource allocation problems to be raised at the beginning of the project while there is still time to do something about any shortfall. Testing will also be a critical participant in initial and on-going Risk Analysis/Management.

Requirements and Specifications – While the requirements are being gathered and written, Testing begins writing a Test Strategy to outline the overall approach to testing and related assumptions and constraints. Testing also reviews the requirements for clarity, completeness, ambiguities, and testability. This means preventing defects from occurring in the code and having to be found during a future testing phase, when it is more expensive to fix.

Design – While Development is documenting an architecture and detailed design for the application, Testing continues Requirements Analysis and begins creating Test Cases (for inclusion in the Test Plan) and any other Test Assets (test coverage matrices, test data, test automation frameworks, etc.). Testing also reviews the design documents for accuracy and testability. Design for testability and the easier all areas of the application can be tested – improving quality while saving time and money.

Coding (unit) – Once coding begins, Testing must work towards completing Test Cases, Test Plans and Test Assets. Identification of Build Verification Test Cases (BVTs) and prioritization of the remaining Test Cases are completed at this point. Testing begins performing some unit testing (developing scripts and automation where applicable), depending on the scope of testing outlined in the Test Plan.

Coding (integration and stabilization) – As integration begins, Testing can begin executing BVT Test Cases as part of “Smoke Testing” or “Stability Testing”. Testing will then move on to high-priority Test Cases in areas where sufficient development has occurred. As integration progresses, Testing is able to test more and more of the application’s functionality, reporting all defects. When the application nears functional completeness, Testing completes one or more test passes (execution of all Test Cases). Finally, focus shifts to system, scenario, stress, and performance testing. If Testing did not have all information available before or the requirements have changed, test execution and creation and/or updating test case steps may have to occur at the same time.

Coding (code freeze and bug fixing) – During this phase, Testing’s primary activity is regression of defect corrections. Testing also participates in the triage of defects (reprioritization to reflect risk, schedule slips, or removal/addition of features) as needed.

User Documentation – Testing reviews all user documentation (manuals, help files, etc.) for accuracy, usability and completeness.

Release – Testing can sometimes be responsible for the final builds that lead up to the release. When the release decision is made, Testing collects and archives all project data, documentation, source and tools. Testing also participates in the project Post Mortem, where problems and solutions encountered in a project are reviewed for learning purposes.

Support and Maintenance – As externally-found defects are reported to the support personnel, defect reports are analyzed by Testing to determine if the defect is reproducible and whether it was known prior to release or not.

Starting test activities early means you can catch small quality problems before they become big quality problems later on. Review plans and designs for weaknesses before the software is ready to be tested. Test the software as soon as it is available and log those bugs all the way through the project.

Posted in  All, Planning for Quality, Test Planning & Strategy | Tagged , | Comments Off on Test Throughout the Development Lifecycle

Data-Driven Testing

One of the risks of any application is being able to do enough testing in the time available. Another common risk is getting at those critical components behind the GUI to ensure that they are as bug free as possible before and after integration with the rest of the system.

Black box testing can tell you whether the GUI layer is functioning as it should, by taking in user input and responding with a result. But what if there is an error, how do we know what layer (GUI, Application, Database, etc) is creating the error? What if our black box tests don’t result in any database or application layer bugs? Does that mean there aren’t any? What if the GUI layer is masking the errors?

The tester needs to avoid the risk of automating a fluctuating GUI and at the same time provide the volume of tests needed to ensure that the application and database layers are stable and functioning.

Test Data Creation

In order to begin performing data driven testing you first need to create good test data. Sometimes sources for this data are available from previous testing efforts, from users of the previous versions of the application, or perhaps obtainable by purchasing test suites from a vendor. More often than not, this test data needs to be created from scratch in the specific context of the application to be tested and the test plans to be executed against it.

Sending the Message

Once the database is populated with your initial data, you will want to take that data and construct a message, command, or request to send to the application that you are testing.

Constructing the message can be as simple as extracting the information from the database and putting it together in the order you need. Or there could be additional, more complex, logic required to construct the message that the application needs to receive. Working with scripts and databases together can give you a very versatile and useful solution.

Getting Your Results

To get the results of sending your message to the application, you need to be able to collect the result codes or expect your application to send back a response. Assuming the latter is the case, your test tool would wait for that response, process it as necessary and then compare aspects of the response to the value in an Expected Results column in your tool’s database. Depending on the result, a Pass or Fail can be determined and a log entry created with as much detail as you care to capture.

Example Testing Problem

To pull all of what we have talked about so far into a tangible example, let us imagine a certain application that we want to test.

The application is a multi-tier web application that passes information from its User Interface layer to its Application Layer, via XML messaging. These XML messages are made up of a number of fields of all different types. The application takes these messages, processes them, and decides what the appropriate response is – return the information requested, perform the required action, etc.

If we assume that a typical message can contain 20 data elements or fields, and each data element needs to be tested with four different values: valid, error, upper field length boundary, and lower field length boundary: we need to perform over ONE TRILLION tests.

4 to the 20th power = 1,099,511,627,776

This number of fields is not unreasonable, and the different values we mentioned above do not include every case that may need to be considered depending on the constraints for the message’s data elements. Are all the fields required? Can they be blank? What about surrounding whitespace characters, are they stripped away for each field? What about fields that take codes like ON/OFF or RED/GREEN/BLUE/BLACK/WHITE? Each field will need a certain number of tests based on its constraints.

This is where you need to make use of Equivalence Classes and other test planning techniques to determine the minimum number of tests for each field. For example, if we can assume that we only need to vary the input of 15 fields out of the 20, and that each field contains only either valid data or invalid data, the number of combinations of test data we now need is much less.

2 to the 15th power = 32,768 <– still a lot of tests

This is obviously a coarse example, but it makes the point. What if you had 30 potential data elements in your message? Generating this amount of data, executing the tests, and reviewing the results takes a huge amount of time.

Always, make sure that you have well designed test cases before undertaking your automated testing. A little upfront analysis and planning can save you a lot of work.

Defining Your Test Tool

Before creating or purchasing any software application or test tool, the first thing one must do is to collect and enumerate the requirements.

In general the tool needs to:

  • Create and/or populate a database based on a predefined schema including an “Expected Results” column that adequately populates the database based on the elements created
  • Create a ‘message’ or formatted string, delimited by tabs or in XML, for sending to a network location, where the application under testing is waiting for connections and incoming requests or data to be delivered
  • Send the message to the network location monitored by the application being tested
  • Listen for a response from the application
  • Compare the response to the value in the appropriate row in the data table
  • Report on the success or failure of the test based on the result of the comparison of the actual response received and the corresponding value in the expected result column

Before stating that the basic requirements are complete, it would be much more useful if the tool could execute thousands or millions of tests. The time for these tests to complete could be extensive. It also may be that the tests should really be run at times of the day when there is low usage of the application or low network traffic. Both of these needs require that the tool is able to run unattended and perhaps is able to start up at a scheduled time.

But remember, if you are building your own tool or even buying one, keep it simple. After all this isn’t the product you are trying to ship. The first priority is to make the test process work better than it is working right now. You can add in other bells and whistles later.

Summary

The framework required for any basic data driven testing tool is made up of three main components: File Input/Output for reading from configuration files and writing reports, a database for storing the test data, and an engine with which to extract the data from the database and make meaningful directives and requests of the system under test.

And that’s that. With a tool comprised of these core components, you can send potentially trillions of messages to an application for testing purposes – far more than you could, or would ever want to do manually.

Posted in  All, Automation & Tools, Test Planning & Strategy | Tagged , , | Comments Off on Data-Driven Testing

Risk Management

Risk management can be defined as creating and maintaining plans for controlling real and perceived risks paired with monitoring the effectiveness of those plans.

DO NOT wait until a risk becomes reality before deciding what to do about it. A software development project contains many elements of uncertainty that manifest themselves as risks. And in most cases it will be too late to do anything about them if they become reality. You end up managing the consequences rather than the risk. Risk management allows you to be active in attempting to predict what could go wrong and in addressing those potential problems as early in the project as possible. Remember Smoky the Bear: You too can prevent forest fires. Or in this case, risks becoming reality.

Risk management needs to start at the beginning of a project to be of benefit. Risks feed into the project plan and help determine how you run your project while trying to address the risks in order of priority. Managing risk early is almost always less costly and painful than cleaning-up after the fact. However, introducing risk management at any point in your project will give you some benefit; it is better late than never.

The process of risk analysis and mitigation is comprised of three phases: risk identification, risk estimation and evaluation, and risk planning. These three phases allow one to identify the risks, judge their potential impact, and plan to avoid or minimize the risks. Identifying risks allows you to evaluate and plan for them, and to be prepared to respond when mitigation strategies fail. Much of learning to judge risks and planning ways to address them comes with experience; it is an acquired skill.

The Risk Management Plan

The Risk Management Plan details how to manage the risks associated with a project. It details the risk management tasks that will be carried out, assigned responsibilities and any additional resources required for the risk management activity. In a smaller scale project, this plan may be simply a “Top 10” risk list that is reviewed at regular project status meetings.

The first step in developing your Risk Management Plan is to brainstorm the current risks.

No matter what stage your project currently is in you need to know the kinds of risk you are faced with now, and most likely will be faced with throughout the rest of the project, before you start trying to decide upon strategies for managing those risks. Create an initial list of risks and use this list to prioritize and monitor the risks throughout the project. The risk list should be reviewed regularly to keep it up to date and to evaluate the effectiveness of risk mitigation strategies. You will find the results of these reviews can drive revisions to the project plan itself.

Next, you have to make someone responsible.

Someone on the project needs to be responsible for collecting and recording risks as they are identified, tracking status, and scheduling review meetings. Also a group of concerned stakeholders should be identified. These stakeholders are drawn from both managerial and technical personnel on the project, and should include the project manager, the leads for the test and development teams, and the product manager or customer representative. This group needs to consistently attend the risk review meetings.

Once you have your initial risks recorded and someone has taken responsibility for keeping track of those risks, identify an attribute of the risk that you can measure for each risk.

You can pair these measures with indicators, or thresholds, that tell you when each risk is about to (or has) become a reality. Tracking these trends will help take the guesswork out of whether your mitigation strategies are working. And they will allow you to see how much room you have. The approach that will be used to monitor and reduce each risk should be clearly documented and accessible to the project team. The approach should also outline a contingency plan in case the risk occurs despite any mitigation efforts.

Finally, you need regular reviews.

It is extremely important to realize that risk management is only effective if it is an on-going activity. All too often a project team will produce the initial risk list, update it once, and then put it on the shelf and forget about it, becoming embroiled in fighting the fire of the day. The Risk Management Plan should outline a schedule for regular risk status reports and risk review meetings.

Posted in  All, Planning for Quality, Risk & Testing, Test Planning & Strategy | Tagged , | Comments Off on Risk Management

Stay on Track in Any Economy: 7 Hot Tips for Better Software Projects

I co-authored this article with Andrea McIntosh and we published it with StickyMinds.com


There are some basic truths and best practices that hold firm regardless of the economic climate in which you find yourself. Tightening up software development and planning practices and using resources more efficiently will always give you a return in time, money, and quality.

1. Minimize Overhead Work During a Project
Stop tinkering with that project plan! Put away those audit checklists! Cut meeting times to the bare minimum and keep them focused: take in-depth or lengthy discussions “off-line” (that is, have them outside of the meeting, where the people directly involved can spend focused time on the issue). You need to ship software, so make it your mantra. Don’t do any activity that makes it harder. To make sure that you ship good software, keep reading…

2. Cut-Cut-Cut Features
Cutting buggy features and excess new functionality to meet release dates will allow you to get revenue in the door sooner rather than later… It is hard to decide what and when to let go. Get help deciding (your customers or partners are good places to start). Prioritization of features will also give your project team more guidance on what the key high-priority items are so they know where to spend their time instead of investing long hours on less important or trivial features.

3. Manage Stakeholders’ Expectations
Work with your stakeholders to make sure that, as things change and schedules slip, they know what’s going on. They can help with determining priorities, and provide valuable input into making decisions. At the same time, they will appreciate you being honest with them about any changes needed or problems encountered.

Asking your stakeholders for input will make them part of the process–and they will have a greater interest in seeing you succeed (as it will ultimately benefit them by giving them what want!). As well, the stakeholders’ might be willing to let certain things go–such as being more flexible on a shipping date in order to keep a feature, or on the feature set in order to meet a release date. It never hurts to ask, all they can say is “No”.

4. Tools Aren’t Silver Bullets
Tools, whether used for planning, requirements management, CM or testing, need to be supported by manual tasks and workflows first, not the other way around. Before buying a tool, consider how well the tool fits into the processes you are currently working with, regardless of whether those processes are formal or informal.

Remember, you need to spend time and money to train your resources to be able to integrate the new tool into your project effectively. If you can’t do that and ship your next release simultaneously, while doing a good job at each, the tool will quickly become shelf-ware and the organization will have lost money, time, and buy-in for future tool purchases.

5. Make Risk Your Guide
Think of risk as your sherpa as you navigate the Himalayas… Proper risk analysis can provide guidance for you and your team in deciding what you must do, what you can avoid, and what you are not going to worry about. Perhaps appoint one person on the project team to be the “Risk Officer”, responsible for tracking the project’s risks and the status of mitigation/avoidance plans, and to report on this information to the rest of the team during project status meetings.

Regular risk reviews (even informally during the project status meeting) and implementation of mitigation strategies will make your software journey a safer, more successful experience.

6. Look for Quick Wins
Quick Wins are things that are easy to implement or adopt and have a large potential return on investment (ROI) in the short term. Continually looking for Quick Wins in a planned manner means you are now doing continuous process improvement at a rate that makes sense for you. Quick wins are the kinds of small changes that you can make in the way you do things that, while consuming less than 5% of your daily available time, can add up to significant savings (time, effort) at the end of the project.

For example, you may find that your test team is spending too much time waiting for builds. You may be able to quickly pull together a script that builds your software automatically before the first person arrives at work in the morning. This would allow you to check to see if the build was successful without wasting any more workable hours.

7. Skip the training–Hire the brains
Bringing in experts in the present can pave the way for leveraging less senior resources in the future. This is one good way to avoid or mitigate risks. These “hired guns” are experts in ramping up quickly on new projects, and can become effective in a very short period of time.

Although this is contrary to the “Never add people to a project late in the game” axiom derived from “The Mythical Man-Month” (Frederick Brooks), if you hire an expert and not a generalist, you can reap great benefits.

Conclusion
In short: keep your product and processes simple, do it well manually first and automate only when it makes sense, make changes and improvements in process as you go rather than all at once, and never stop looking for better ways to do things that make sense for your organization, product, and market.

Posted in  All, Business of Testing, Other, Planning for Quality | Tagged , | Comments Off on Stay on Track in Any Economy: 7 Hot Tips for Better Software Projects

What is Risk in Software Development?

Risk is often defined as the combination of the likelihood of a problem occurring and the impact of the problem if it were to occur.

  • Risk concerns future happenings.
  • Risk involves change in mind, opinion, actions, or places.
  • Risk involves choice and all the uncertainty that that choice itself entails.

Risk is something with which we are all somewhat familiar. We know that it is risky to jump out of planes, or invest money in the stock market. But we don’t always think of software development as risky – particularly if our software is not mission critical (that is, people’s lives don’t depend on it). While insurance companies and actuarial analysts spend most of their days dealing with risk, software development teams often spend a mere fraction of their project schedule thinking about risk.

In this series, we introduce you to some basic concepts and principles of risk management, and how these concepts and principles can be applied to software development projects to increase the probability of success.

In everyday life a risk is an exposure to the chance of injury or loss – a thing or course of action involving uncertainty and therefore danger. A risk, in the software industry, can be anything that may threaten the successful completion of a project: increased costs, delayed completion, loss of quality, or loss of market-share.

Two factors can be used to determine the magnitude of a risk: likelihood of occurrence, and impact of occurrence. When combined into a single indicator the risk can be judged relative to other risks as High, Moderate, or Low.

Many organizations still do not identify risks at the beginning of a project, or review and update those risks throughout the course of the product lifecycle. Estimating and planning is done as if all variables are known, and outcomes certain.

The reality is that risk results from changes in development personnel, market conditions, customer expectations, and even business conditions for the development organization.

Experience shows that the large majority of risks have a direct or indirect impact on schedule, and therefore implicitly on cost. Some have a direct impact on cost. The rest may have no direct impact on cost or schedule, but affect other areas of the project such as quality.

Some common risks include: the technical requirements of the product or system, constraints placed upon the project or product by the customer(s), budget, or schedule.

Types of Risks

It’s important to distinguish between direct and indirect risks. A direct risk is one over which some degree of control is possible whereas an indirect risk is one that cannot be controlled.

While one should not be unaware of the indirect risks, they are of little consequence in a practical sense: since one cannot change them, there is little to be gained by worrying about them.

Sometimes, an indirect risk may actually be a direct risk. For example, you may be dependent on an external supplier for a component or platform for your product. This appears to be a risk that cannot be controlled. But, by having contingency plans for those components, you can take control of the risk: develop a relationship with the supplier so they listen to your needs, choose alternate suppliers, or choose to develop the needed components yourself.

With indirect risks, you either have to gain some degree of control, or you need to just make note of them. Having them documented will allow you to easily revisit the risk and the decisions that were made as necessary.

To help determine if your project would benefit from Risk Management, look at your project and ask yourself the following questions.

  • Is the project scope firm or does the scope keep expanding?
  • Are estimates accurate?
  • Is there time to “do it right”?
  • Has the technology been proven?
  • Are the users of the system experienced with the type of system being developed?

These are just a few of the questions to ask yourself when looking for potential risks related to your project. If your answer was “No” or “Unknown” to any of them, it means that you are facing at least one risk to your project being a success.

Posted in  All, Planning for Quality, Risk & Testing, Test Planning & Strategy | Tagged , | Comments Off on What is Risk in Software Development?

eXtreme Testing – Ramp Up and Keep Up!

eXtreme Programming, or ‘XP’ as it is also known, is a relatively new development methodology (~4 years old), which treats software development as a ‘jigsaw’ puzzle: a product is slowly built by adding small pieces to a larger body of work. The methodology is designed to deliver the software that the customer needs when it is needed, accommodating changing requirements, even late in the life cycle.

The four sought-after values in eXtreme Programming are: Communication, Simplicity, Feedback, and Courage. Testing needs to support Development in these values to allow them to move forward with what they have to do, which is to implement the current iteration’s functionality. Development needs to know that collateral damage caused in the current iteration will be caught immediately, before it is time to release.

The goal of this article is not to talk about XP… directly – there are lots of other articles and sites out there about eXtreme Programming that you can read. This site is about how to test software that is developed following XP principles; using a methodology we can call eXtreme Testing.

The biggest challenge of testing, regardless of the methodology, has always been to cover everything with the resources and time available. The intent of eXtreme Testing is to directly address this issue but in the framework of eXtreme Programming. eXtreme Testing is designed to allow testers to get ramped up on the project as quickly as possible and then to allow them to keep up after that. The values of eXtreme Programming are directly applicable to Xtreme Testing.

The general approach of eXtreme Testing is to define a straight-forward test strategy for the iteration. Prioritize the acceptance tests (ATs) defined by the client and any complementary tests necessary. Automate what can be automated. And, ensure that each release has its new functionality ‘Smoke Tested’ and that no regression in previously existing functionality has occurred. The test activities must be efficient, well structured, easy to run, and the results easy to communicate. The focus of eXtreme Testing is to be confident that what needs to be tested has been without slowing the project.

You want to try eXtreme Testing on your project? Add a little to your current methodology or try it all at once. The basic principles described here can benefit any project, XP or not.

Workflow

The eXtreme Testing workflow is seen to ‘lag’ slightly behind the development cycle. Thus, if a project is estimated to have 10 iterations of development, there will need to be (a minimum of) 10 iterations of testing (test planning, automation, test), but they may lag by up to one iteration, making the complete project length 11 iterations.

NOTE: Depending on the type of project, full automation of testing may not make sense or even be possible.

The eXtreme Testing workflow is comprised of several parallel streams of activity (A to F in the diagram below), including Project Planning and Development:

  1. In each iteration, the development team and the client agree to the features or stories (A) that will be implemented and delivered by the end of the iteration. Out of this negotiation, the client will define the ATs for the coming release.
  2. Testing will perform a high-level test analysis of the stories and incorporate that analysis and the client’s ATs into a simple test plan (C) for the iteration.
    NOTE: The test analysis, when presented to the client, may result in modification of the ATs.
    NOTE: In the initial iteration, more thought and effort must be put into developing the test strategy than with successive iterations, much the same as when defining the project architecture.
  3. As part of the test planning component of the iteration, estimates of effort required to automate the ATs and execute functional testing are created and compared against estimates from previous iterations. The test effort is then prioritized to accomplish as much AT automation and functional testing as possible within an iteration’s timeframe. The remaining tests will be addressed at the beginning of the next iteration.
  4. While development (B) of the current iteration’s functionality is being undertaken, automation of the ATs (D) of the current iteration’s developed functionality is started (in order of descending priority), and automation of any unfinished or defective ATs from the previous iteration is completed. The early automation of new ATs will allow defects to be uncovered and corrected on an on-going basis.
    NOTE: The automation of the new ATs may not all be available until after the smoke test for the current iteration, but *some* of the automation could be run on the current iteration and the remainder of the tests executed manually.
  5. While development of the current iteration’s functionality is being completed, functional testing (E) of the previous iteration’s developed functionality and regression of corrected defects is on-going. The problem reports resulting from this testing will be prioritized and addressed accordingly.
    NOTE: Some problem reports could be postponed until the next iteration or perhaps even longer.
  6. Integration of the current iteration’s functionality should be followed by a Smoke Test of the new functionality paired with regression testing of corrected defects. This allows a sanity check of the integrated components and how they are working together. The Smoke Test will be comprised mainly of the ATs for the current iteration and critical ATs from previous iterations.
    NOTE: Complete functional and regression testing will occur at the beginning of the next iteration.
  7. Release (F) of the current, tested, iteration is made to the customer. This small release is designed to generate feedback on the requirements and functionality as implemented to date and is used as an input to the Feature Set Definition component of the next iteration.
  8. Re-Iterate.
Integrating Testing with eXtreme Programming

Integrating Testing with eXtreme Programming

What about other sorts of testing?

Performance, Stress, and Usability testing are just a few of the other sorts of tests that may be appropriate to the project. Time and resources need to be scheduled to undertake these activities at sensible points in the project life cycle. In some cases it may make sense to devote an iteration to these types of tests while Development takes the opportunity to catch up on outstanding problem reports.

How does eXtreme Testing meet the ‘rules’ of eXtreme Programming?

The basic set of testing rules for eXtreme Programming are covered by the following rules of eXtreme Testing:

  • Unit Tests must be documented for each component
  • All Acceptance Tests (ATs) must be documented
  • Uncovered defects must have Regression Tests created to guard against its return
  • All Unit Tests must pass before a component is allowed to be integrated
  • All ATs and Regression Tests must pass before a Release is made
  • Automate any test when it makes sense and where it is possible

More on eXtreme Programming?

Where can you get more information on eXtreme Programming? There are links to projects, conferences, books, web sites, and people that can be of help.

Posted in  All, Agile Testing, Automation & Tools, Test Planning & Strategy | Tagged , , , , | Comments Off on eXtreme Testing – Ramp Up and Keep Up!