Know Yourself as a Tester

Posted on January 3, 2011 by Christin Wiedemann

Who are you? How do you behave? And how does it affect your testing?

I was pair-testing with a colleague, which gave me an excellent opportunity to study his testing behaviour. He had to enter a five digit number repeatedly, and in this particular test it was irrelevant which number was entered. To my fascination I noticed that nine times out of ten he would enter “12458”.  Of course I had to ask what was so special about “12458”, and it turned out he was completely unaware of his behaviour – on the contrary, he was convinced he was entering random numbers!

To me, this served as a bit of a wake-up call, making me contemplate my own behaviour and how it affects my testing in general and my exploratory testing in particular. Do I to have favourite numbers that I tend to enter more often? Am I more prone to using keyboard short-cuts than the mouse pointer to navigate? How often do I think I am exploring new, untrodden, trails when in reality I am just walking down the same well-beaten path as always, and might almost as well have run a scripted test?

Here session reports in the spirit of SBTM work as a support for me, helping me to be more creative and my testing to be more exploratory. By making short notes when I do exploratory testing it is easier for me to identify patterns in my testing. These behaviours are not necessarily bad, but I believe I need to be aware of them. Furthermore, I use my old reports as inspiration – or “seeds” – for the next session.

Everyone has little quirks, even when testing, and I think it is important that we are aware of them.

Posted in  All, Agile Testing | Tagged , , | Comments Off on Know Yourself as a Tester

Tying Up Some Loose Ends

Posted on January 3, 2011 by Christin Wiedemann

I think it is time for a follow-up on my experience of applying Thread-Based Test Management (TBTM). In October last year I blogged about my attempt at setting up my tests for a new project using TBTM.

It was a smaller project, with me as the only tester which gave me a lot of freedom to try new things, and I actually did what I set out to do! First I created my mind map, or fabric. I added all test activities I could think of to the fabric as threads. Using different colours and icons I visualised the priority of the threads. The first version of the fabric as it looked before I started testing I kept as my test plan, and this I sent to the external customer. To my joy the mind map version of a test plan was quite well received!

TBTM

Thereafter I started testing. I would typically be working on two or three threads at the same time, and I would not “tie off” or finish a thread, but rather do some testing and then come back to the same thread later, after having worked on some other threads too. This way I ended up digging into the details of most threads at the same time, rather than focusing on one test task and “completing” it before moving on. As my testing progressed I kept updating the fabric. The visual presentation of the current status turned out not only to be a great help to me, but also to the developer I worked with. In the end, he would actually only use the fabric to see what work remained and not our usual defect reporting system.

I had planned to write daily status reports, but instead I actually ran my tests as unfinished, not time-boxed SBTM inspired sessions and wrote session reports. It turned out I needed the reports to keep track of what I was doing. For test charters, I drew a lot of activity diagrams.

When all testing I had time to do was done I had a fabric with a lot of green threads and unfortunately some red too. I took my “final” fabric and used it as my test report. Together with some clarifying notes it constituted all test documentation the external customer received.

My conclusion after this experiment is that for me personally I think a hybrid of SBTM and TBTM is the way to go. I do feel the need to work on things in parallel whereas I usually do not feel the need to time-box, but on the other hand I really like my session reports. I will continue to work like this – it has been exploratory, controlled (in the good sense) and well documented (in the sense of “as needed”). Oh, and I left very few defects for the customer to find and they were all of low priority…

Posted in  All, Agile Testing, Test Planning & Strategy | Tagged , , , | Comments Off on Tying Up Some Loose Ends

Regression Testing – Strategic and Risk-driven, Can You Afford Not To?

Posted on November 23, 2010 by Trevor Atkins

Planning your testing effort for the next project: How much time do you have?  How many people do you have?  _How_ big was that scope again?

As testing is virtually always constrained with not enough time, not enough people, not enough infrastructure, etc it is vital that the testing effort is able to prioritize what is critical and what is not so the scarce resources can be applied in the most valuable manner possible.

New functionality is where the excitement is, and definitely where there is a high probability of defects.  And in the face of tight budgets and schedules, even new functionality testing is often short-changed.

But is it really the only place of high risk? How many times has the code been changed and something seemed to break somewhere else?  What if the impact is a data integrity issue?  What if the collateral damage is deep into a different business process?  What if performance of key tasks degrades after making a “small” change in a supposedly unrelated area of the system? Continue reading →

Posted in  All, Business of Testing, Planning for Quality, Test Planning & Strategy | Tagged , , | Comments Off on Regression Testing – Strategic and Risk-driven, Can You Afford Not To?

Click OK to Crash

Posted on November 3, 2010 by Christin Wiedemann

I – of course – use a risk-based approach in my testing, meaning that checking the exact phrasing of error messages is very far down on my prioritised list of test tasks. As long as the message is relevant and understandable I am happy enough.

I have an old favourite error message that it is hard to compete with. A couple of years ago I was using a free software that was really fantastic, but would occasionally and unpredictably crash, in which case the simple yet suitable error message ‘Click OK to crash’ was shown. There was an OK button, the only thing you could do was to click it, and yes – the program crashed.

The error message completely failed to give me any clues as to why the program crashed, but it certainly told me very clearly what was going to happen, and it put a smile on my lips, even though I would loose my unsaved work. In my opinion simplicity in error messages is definitely preferable, unless you are actually going to put some useful information in there. Do not try to obscure the fact that you messed up by using tech lingo or providing a pointless novel.

I also remember painfully well working on my thesis late one night when I encountered the king of all error messages: Kernel panic. No ambiguity there – I panicked too.

If there is a point hiding in here it is this: error messages are important, spend some time on them. Think usability, not debugging.

Posted in  All, Planning for Quality | Tagged , , | Comments Off on Click OK to Crash

Physicists – Testers in Disguise?

Posted on October 28, 2010 by Christin Wiedemann

My background is in science. I have spent 11 years (a third of my life, believe it or not) studying mathematics, statistics and most importantly – physics, experimental astroparticle physics to be specific.

I have been trained

  • to be sceptical
  • to question
  • to think analytically
  • to think logically
  • to be curious
  • to try to understand how things work rather than accepting stated facts
  • to explore

All of these things I think make very good qualities for a tester too.

My research consisted of searching for a signal in a data set made up mainly of background noise. Feel free to read my thesis. In order to do my research I had to write my own software. Since the results of using my software to process the data were going into my thesis I had to test that the software was behaving as I expected it to in an attempt (futile maybe) to minimize the risk of making a complete fool of myself.

I claim that testing in a wider meaning of the word comes naturally to experimental physicists, even when talking about software testing. The life of any experimental physicist consists of

  • data acquisition
  • data analysis, more often than not using some homemade software
  • publishing results from data analysis

Publishing (preferably interesting) results is the basis of your career, if you do not publish you do not exist. Imagine what would happen (and does happen) if you publish results you later have to retract because your software is found to have severe defects. Physicists are aware of what is at stake – and unlike what is generally the case in the software industry, every mistake is going to hurt the physicist personally.

Hence physicists – and all other scientists with integrity – test their software tools meticulously to make sure they understand how they work and that  they work as expected. It is not a strict, structured testing that ISTQB would approve of, but the physicists have their hearts in the right place. They want things to work and be reliable, and is that not really just what we all want?

Posted in  All, Other | Tagged , , | Comments Off on Physicists – Testers in Disguise?

Spinning Threads Into Yarn

Posted on October 21, 2010 by Christin Wiedemann

Recently the approach I have had to my testing has been heavily influenced by session-based test management. I have made a test plan consisting of a high-level list of test tasks. The testing has been exploratory, performed in sessions on a given topic, e.g. a function. I have two problems with this:

  1. As much as I like lists, they make bad test plans – at least to me. There are always too many tasks so the list will be too long, covering several pages in a document, making it hard to get an overview. It is also difficult to depict relationships. I have tried different groupings and headers, and managed to create nightmare layouts that are impossible to read. A list is also highly binary – either the task is done or not, there is no equivalent to “work in progress”.
  2. I would rarely be able to finish a session without interruption. Something urgent would come up and I would have to abort the session, and when restarting the conditions might have changed. As I discussed in the post on October 17th, I also tended to feel obliged to completing the session before I took on a new task, even though there might have been more important matters surfacing after the session started. In this situation it was of course hard to keep track of the status of the tasks.

The appeal of thread-based test management is of course that I can perform test tasks in parallel – it is not necessary to say that a test task is done. Instead I can scrape the surface of everything once and sort of work my way down to the insignificant details from there.

I have resolved to use a different approach for the next test period. This is what I have done so far:

  • I have installed the open source mind map tool FreeMind
  • I have created a mind map (I call it fabric) with
    • Error reports
    • Product heuristics
    • Generic heuristics
  • Since the fabric only contains short thread names, I have introduced knots that are (k)notes in simple text format that I link, or tie, onto the threads. The notes contain additional informtion such as hints, tips and reminders.
  • I have compiled the stitch guide. The stitch guide provides guidelines on how I think that my project should use thread-based test management. The guidelines are not rules, but suggestions intended to promote consistency.
  • I have a template for daily status reports. The report can contain anything I feel needs writing down during the day, but should at least contain the names of the threads that have been tested in some way. I am currently looking for a more fun name that “Daily status report”.
  • The actual testing will of course be exploratory.
    Fabric Mindmap
    A fabric. Colours and icons are used to show priorities and status. The red arrow indicates that there is a knot tied onto the thread.

My plan is to use the first version of the fabric as my test plan. During the test period I will keep updating the fabric, and at the end of the test period the current status of the fabric will be my test report.

Note to the reader: This is my current interpretation of thread-based test management, and this constitutes my starting point. Hopefully it will evolve into something that I find useful, and turn out ot be an improvement compared to today. If not I will not hesitate to chuck it and try something new. I have big hopes though and cannot wait to get started!

Posted in  All, Agile Testing, Test Planning & Strategy | Tagged , , | Comments Off on Spinning Threads Into Yarn

Picking Up a New Thread

Posted on October 17, 2010 by Christin Wiedemann

I have just spent the weekend at a very inspiring peer conference on exploratory testing (Swedish Workshop on Exploratory Testing, SWET1) in Stockholm.

There were many interesting presentations and discussions, but what is on my mind right now is James Bach’s presentation on thread-based test management, http://www.satisfice.com/blog/archives/503 . I have been trying to adopt Session-Based Test Management (SBTM) for a while, but never managed to do any proper time-boxing since I would typically be interrupted in the middle of a session and have to abort or restart.

Quite often I will start a test activity, be interrupted and not finish the session, start a new test activity, not complete that session either for whatever reason and so on. Working this way makes me stressed since

  • It feels like I never finish anything.
  • I do not have an overview of what I am doing and what the status of the different tasks is.

I have also come to realize that in some cases I feel a bit hemmed in by the actual session. Even if the conditions change or something urgent comes up I still feel obliged to finish the session before I start a new activity. In those cases when I work in a chaos of sorts I think this perceived need to “be loyal” to my session reduces my efficiency.

So, I’m going to have a go at thread-based test management instead, and I start by making a mind map!

Posted in  All, Agile Testing | Tagged , , | Comments Off on Picking Up a New Thread

Play Me a Song

Posted on September 13, 2010 by Adam Geras

My sister was visiting us a few weeks ago so that she might meet her new niece in person. She lives about 1500km away from us, so it’s just far enough away to make these sorts of visits infrequent.

I learned a lot.

I watched her reverse-engineer a song that I’d been singing to my son and reduced it to its essentials so that I might learn to play it on the piano. In about a minute and a half, she identified the major chord and said, “just play around with that chord with the right rhythm and you’ll get it”.

I didn’t believe it.

That is, until I started playing the song, a couple of weeks after she left. Her gift – countless hours of practice as a performer and teacher that made it possible notwithstanding – was to show me the essentials, guide me with just enough to get started, and then to let me go experiment for myself.

I didn’t get a lesson plan.

She didn’t give me a script, a transcribed version of the song into notes on the paper. She didn’t tell me the rhythm, she let me feel that out for myself. She didn’t tell me what to do when.

Turns out this is what checklists should do for testers, especially those business testers that are not necessarily software test professionals in the first place. They know their business. They know how to use a computer. They need guidance, but as I didn’t need a script to learn a song on the piano, they don’t need a script to test effectively.

In my experience, the more we tell our testers what to do, how to do it, in what order, how often, etc., the more likely it is that they will do just that. And only that. Sticking to script is rarely a good thing in testing.

Checklist items are chords, not notes. If you believe the tester is a person, a sentient being capable of putting emotional labour and thought into their work, then identifying the chords is enough. They will play the notes without you going through the effort of transcribing them.

Give them room to create their own art.

Posted in  All, Agile Testing, Test Planning & Strategy | Tagged , , , , , , , , , | Comments Off on Play Me a Song

Survey Report – Project Success & Test Outsourcing

Posted on September 9, 2010 by Trevor Atkins

This summer, we put out a quick 10-question survey regarding software development and outsourcing or contracting of testing services.

The intention of the survey was to use the responses to paint a picture of the attitude or approach towards outsourcing of testing in comparison to how organizations seek success for their projects and teams.

In general it was hoped that the responses might indicate that:

  • Organizations were looking beyond hourly rates to the total value that was being delivered by contract service providers (eg: productivity, increased capability/capacity, organizational planning and execution improvements, bottom-line impact).
  • Organizations were seeking solutions that: were more than just bodies to add to a project team; provided strategic thought leadership rather than just execution on assigned tasks; built a relationship with a partner specialized in delivering specific value-adds.
  • Organizations were open to looking beyond the confines of their own offices to find much needed support such that not all of the team members needed to be co-located in order to be effective.
  • Organizations were extracting the benefit of up-front thinking about risks and constraint trade-offs by investing into test strategy and effort estimation planning activities.

Not all of these expectations were fully supported in the responses, suggesting that there is still significant opportunity for improvement benefits. However the responses provided a picture of a stronger focus on quality and testing activities than would have been expected 10 years ago.

Read the full article…

Posted in  All, Business of Testing, Other, Planning for Quality | Tagged , , , | Comments Off on Survey Report – Project Success & Test Outsourcing

Using Agile Principles to Get an “A” With the Customer Every Time

Posted on July 9, 2010 by Trevor Atkins

Agile is a frequent topic in process improvement discussions and networking events alike – some going for Agile all the way and others rejecting it as something that won’t work for their projects or customers.

In general, I have found that a significant degree of value can be added to an organization’s software development process and bottom-line through the application of the expressed intentions of the Agile approach.

Consider:

Principles behind the Agile Manifesto
We follow these principles:
Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.
…
 http://agilemanifesto.org/principles.html

If one understands Agile as not a methodology or process but as a philosophical approach to trying to do better at something that has traditionally not worked as well as it could, substantial opportunity exists for improvement:

“This year’s results show a marked decrease in project success rates, with 32% of all projects succeeding which are delivered on time, on budget, with required features and functions, 44% were challenged which are late, over budget, and/or with less than the required features and functions and 24% failed which are cancelled prior to completion or delivered and never used.”, Jim Johnson, chairman of The Standish Group, CHAOS Summary 2009 – http://standishgroup.com/newsroom/chaos_2009.php

This is not to say that using Agile will overcome all the challenges inherent in your development methodology, but rather that using each of the principles included in the Agile Manifesto to change your approach to planning and executing projects can get you significant “Quick Wins”.

For example with the principle: “…satisfy the customer through early and continuous delivery of valuable software”:

  • Organize your project with 5 or more ‘releases’ to the customer for acceptance, rather than just the typical final release at the end.
  • Distribute your effort and functionality evenly across the releases being sure to be able to show useful, increasing value with each release.
  • Have the customer sign-off on each release and ideally start using it in their business activities in order to start realizing the benefits of the investment as soon as possible.

This approach can bring all sorts of benefits in terms of lowering the Total Cost of Quality, but the one key performance indicator we shall make an observation on is the aspect of customer satisfaction.

agileroicurves

By the time the Final Release is given to the customer for acceptance, the business should have already accepted and be ramped up on the previous releases. Therefore the project team already has an 80% acceptance rating going into this last cycle.

It may be a little light-hearted to look at this outcome as similar to entering the final exam with an “A” already guaranteed, but fundamentally the team has reached the end of the project proving with each release that they are in synch with the customer’s needs and are delivering the solution that fits that need.

Good grades aren’t everything when looking for a successful project, but without a happy customer…

Posted in  All, Agile Testing, Test Planning & Strategy | Tagged , , | Comments Off on Using Agile Principles to Get an “A” With the Customer Every Time