Sharing an answer to a couple of recent customer queries in hopes that it is generally useful as well:
The ability to give early feedback during testing is a cornerstone to providing value on a software project. A well designed smoke test is one part of a strategy that can enable that feedback.
How do you determine what to put in your smoke test? First, ask what are the critical behaviours that need to be working. Then consider the types of changes that have occurred since the last test cycle and what you need to check to gain necessary confidence that these things and what they impact are in “working order”. Finally, prioritize which other areas of the system you need to review in more detail than others (eg: what would you do next if you had some extra time).
In general, you should plan to execute the “breadth” of “breadth and depth” testing in a smoke test, reporting readiness and severe issues continuously as you go. For example:
- The first pass of the system should make sure that all the things that you want to test in this cycle are in fact testable (ie: can be accessed and can perform at least one ‘happy path’ behaviour or task). The time to complete this first pass should take you and/or your team no more than 30 minutes – half of that would be better.
- The second pass should be a more detailed investigation into any new functionality. Make sure that whatever has been promised to the customer or end-user in this release has been evaluated such that you can provide early feedback on whether features are working well or not (don’t want to find those issues late in the test cycle).
- The third pass would include regression tests of the high priority behaviours that you know were working last time; things that cannot be allowed to break from release to release.
Successive passes, outside of the smoke test, would then go deeper into riskier areas; areas that have had a lot of change, are relatively more complex, are business critical ($$$), have been problem areas in the past, or haven’t been tested in “depth” yet.
Testing would continue to delve deeper following these priorities, guided by the Test Strategy, as time/budget and resources permit — it is very unlikely you will have all the time you feel you need.
Note: Do not constrain yourself to documented detailed test cases in your smoke test. Utilize checklists, matrices, use cases, user stories, etc.
Make sure to customize the above outline to fit with the unique balance of quality requirements, scope, and constraints you need for your project.