What is Dendrogram-Based Testing? Well, what is a dendrogram to start with?

A dendrogram is a tree diagram that visualises hierarchical clustering. If that didn’t help, a dendrogram basically groups objects in a tree view based on how similar they are. The closer the objects are drawn, the more similar they are.

Thanks for the maths lesson, but how is that useful in testing?

Good question. I’ll come back with a final conclusion later in this post, but I can think of two uses for dendrograms:

Clustering defects: Visually show how similar the defects previously found are.

Clustering test charters (test cases): Visually show how similar planned test charters (or test cases) are.

In order to create dendrograms we need the objects, e.g. defects, to have such properties that we can measure distances between them. This is where it starts getting tricky – how do we measure the distance between two defects? The simplest thing to do is to think of properties we believe to be important and then assign them numeric values.

One example could be the property “User” (P1) and we could assign a defect a value between 0 and 5 for this property depending on how affected we think the user is by this bug. Another property could be “Performance” (P2) or “Business” (P3). Imagine we are testing a web shop and have two defects:

B1:The “This is a gift” checkbox is missing in the GUI.
B2:Memory issue that slows shopping down when you have more than 10 items in your cart.

Each of the two bugs have the properties P1, P2 and P3, and we might to assign values as follows:

B1: P1=5, P2=0, P3=2
(the user is affected, the performance is not but the business flow is also affected)
B2: P1=3, P2=5, P3=0
(some users will be affected, the performance is affected, the business flow is not affected)

Based on these properties we can now see how similar the defects are in a dendrogram. In my earlier post I explained how to create a defect dendrogram with simple example, and I’m not going to repeat that.

Similarily we can assign test charters or test cases properties and create dendrograms. Here properties could be which actors, functions or areas that are involved, and the dendrogram shows a kind of test coverage. If all test charters are grouped together, they test very similar things.

So how do we base our testing on dendrograms?

A defect dendrogram would of course be used to decide where to focus testing. I think isolated defects would be my priority. A single defect far away from all other defects seems too unlikely, maybe there are more hiding that need to be discovered. Then again, if a large number of defects are very similar there is reason to believe that area requires special attention.

A test charter dendrogram would of course be used to help decide which charters to add. A single isolated test charter might be ok for a low-risk area, but might also be a warning flag.

Is this useful?

I have some serious doubts. Firstly, we need to find useful properties and assign them subjective values. The dendrogram will be based on those values and nothing else, so there is a huge risk of bias. Secondly, I have yet to find a good tool to use to draw dendrograms. With more than three variables (defects/test charters) and two or more properties it cannot be done by hand. Of course, writing your own tool would not be too complicated.

Right now I don’t think the value gained outweighs the effort needed. I’m very interesting in hearing arguments that I’m wrong though.

Friday afternoon I was looking through the latest tweets when my eye was caught by the phrase Dendogram-Based Testing. I like all words that have a Greek origin and sound like science, so I had a closer look, and of course it was James Bach introducing a new concept. One that – as far as I understand – is still pretty much missing a definition. No reason to let a small detail like that stop you, right?

After reading up on dendograms I realized that I have actually used them before, but didn’t know they were called dendograms. A dendogram is basically a way to take data points and cluster them based on their properties. They are commonly used in computational biology, and that’s were I encountered them. About a year ago I spent a week of my vacation making dendograms from genome data.

In testing, one way to use dendograms would be to cluster defects. In order to do this you would need to define a set of properties for each defect, and based on these properties it would be possible to calculate distances between the defects and cluster them in a dendogram. I will save the discussion on whether this is useful or not for later.

Example: The android game SuperTester

To the best of my knowledge this game does not exist, but if it did it would be a game in which the player has to find critical bugs in imaginary applications under time pressure. Just for the record, I haven’t thought too much about this so I’m just making it up as I go along.

Let’s say five bugs have been found when testing the actual SuperTester game:

1. Can’t save game (D1)
2. Can’t change sound volume (D2)
3. It is possible to register the same bug twice (D3)
4. Application crahses if you find exactly 13 bugs (D4)
5. Application crashes if you play for more than 59 minutes and 59 seconds (D5)

Now we need to assign these defects properties in order to cluster them. This is the tricky part and requires some careful thinking, Which properties you pick decide what information you will get out of the dendogram. For now I’m just going to pick two properties for the sake of creating an example,

1. Frequency of occurrence on a scale 1-10, where 1 is rarely and 10 often (P1)
2. Severity of defect on a scale 1-10, where 1 is not severe and 10 is very severe (P2)

Time to make a table:

Defects with assigned properties P1 and P2

Ok, the values might not make so much sense but let’s ignore that for now. We have everything we need to calculate the distances. Note that all this assumes that the properties are numeric, if you have other properties such as “red” or “green” you need to decide how to calculate the distance between “red” and “green”. For numbers we use the Euclidean distance. I’m not going to go through all the boring details, but the distance table will look like:

Distance table

D2 and D4 are closer to each other – that is, more similar – than any other defects. Hence we cluster them in cluster A. And so it goes on. What we end up with is the following dendogram:

Dendogram. Defects are clustered based properties

That’s it! Two important points here are i) you need a tool because you definitely do NOT want to do this by hand and ii) how about non-numeric properties, how do you measure distances? I definitely think dendograms can be useful, but a tool needs to be found and then there must be some thought on which properties to use – what kind of information do we want from the dendogram?

This post is just a collection of initial thoughts and is focused on what a dendogram is. I will now crawl back under the rock I came from and think more about how to actually use it for testing.