Can metrics improve agile testing? Measure, Analyse, Adjust

As a tester in an agile project, I want to make visible how I spend my time, so my team can act accordingly and stakeholders know what to expect.

As you’ve probably experienced, plans and estimates are revised during a sprint. I have never heard of any healthy project that has not done that. In an unhealthy project, stakeholders may stick to a loosely estimated initial plan because they do not know better and cannot act on any historic data

Consider a scenario with a release date getting closer. A stakeholder, who is restrained by a contractual budget and a promise of timely releases, asks the team when they are done testing the stories. The team can’t answer with any acceptable margin, so the stakeholder start drilling down into the stories asking specific questions about how many hours are needed for exploratory testing, test design, and GUI automation – basically taking over the test strategy. At the end of the meeting the testers have committed to some arbitrary forced number of hours for each type of testing on the remaining user stories. The only benefit of that is that the stakeholder is satisfied until it is discovered that none of the new estimates are useful at the next status meeting.

In this article I provide an answer to the question: Can metrics improve agile testing. The best way to answer that is to split it into these four sub topics which I will answer in sequence:

  • Why collecting metrics is helpful
  • What metrics should be gathered during a sprint?
  • What is the easiest way to gather metrics during a sprint?
  • How can metrics help improve testing?

Everyone must know why they are collecting metrics

Imagine recording data without knowing what it is used for and why it is gathered in the first place. The first thing I could imagine myself doing was – consciously or unconsciously – making up reasons. Adding to that, I would become insecure if the metrics are in any way connected to my performance on the team. Building on that and the uncertain reasons behind the metrics, some might even alter those in a direction that would be of no benefit to anyone.

Say, for example, that you are recording how much time you spend on testing a story. You have no idea what the data is used for. You want yourself and the project to look good in the statistics, so you start rounding the numbers down here and there – just a tad. Over the duration of the sprint, that become several hours. Then it is time for stacking the backlog for the next sprint and guess what more stories are taken in than you and your team can test within the sprint.

So be transparent about why and how metrics are gathered. Be vocal about why you measure any performance on the team and project. If you are in any doubt of why you are collecting a metric, ask about it and discuss what it can be used for and how it can benefit the project.

What metrics should be gathered during a sprint?

In other words: What does testers spend their time on? The answer is in my experience always some version of these three things:

  • Testing (including running tests against the application under test)
  • Investigating, reporting, and re-testing bugs (including collaborating with coders and product owners)
  • Setting up and waiting for environments or other resources such as test data (including designing scripted test cases based on notes from previous ET-sessions)

One of the most efficient test techniques in agile testing is exploratory testing because there is less preparation and setup required compared to scripted tests. It is not always easy to gather metrics while focussing your energy on finding creative ways to test the system, so you will need a simple system for doing so.

What is the easiest way to gather metrics during a sprint?

One such system I’ve been using for gathering metrics during exploratory testing sessions is a light version of session-based test management or SBTM, which was developed by Jon and James Bach. The core idea is simply to take notes of what you are doing during a time boxed session and mark each entry. The notes have to really simple so it does not take up all your creative testing energy.

The wikipedia entry defines session based testing like so:

Session-based testing is a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting.

A genius of the tool is that you will take specific note of how much time you use on
1) Testing, 2) investigating Bugs, and 3) Set up. These are also called TBS Metrics. When you gather the metrics, you can see where you spend your time, so if you see less time testing than investigating bugs or even setting up the environment, you can take action on for example getting the environment setup automated. If e.g. you spend a lot of time investigating bugs, perhaps the programmers should be more involved in testing or the user stories should be analysed with the product owner to get more specific acceptance criteria.

The important point is that if you collect these 3 simple TBS metrics, you have come a long way towards being able to analyse your results, you can continuously improve your testing effort based on those numbers.

The full SBTM may be a big bite to start off with, so I recommend that you start with SBT Lite which was defined by my mentor Sam Kalman who worked with Jon Bach at Quardev. SBT Lite is much more free-form, but is still able capture the metrics you need.

In practise, a tester would take notes on his/her actions during an exploratory test session.
Before the session starts, a session charter is chosen and noted on the session sheet along with the esters name. The charter could be one of the acceptance criteria in a user story or an entire user story depending on the size and scope of them. A meta session charter could also be to come up with session charters for a set of stories. The charter is basically the mission of the session.

During the session, the tester marks his/her notes as T, B, or S, along with a time-stamp and a one-line description of what happened. The result should look much like a commit log from healthy pair programming session.

When analysing the session logs, it would be evident how much time is used on testing, bug, or setup time respectively. In the full SBTM resource there is a link to Perl tool that can analyse session notes if they are formatted in a specific way, but you should make your own session log parser if you prefer to follow the simpler version or even design your own set of metrics (which I recommend once you’ve tried this for a few sessions).

How can metrics help improve testing?

As I suggested in the scenario in the introduction, it is not only managers and stakeholders that benefit from the collected metrics. Testers can benefit on a personal (professional) level, the team can benefit, if there is a cross-team test organisation that too can benefit from the TBS metrics.

The individual can use the analysis of the metrics to hone his/her skills. Perhaps compare how much time is spent on each task type compared to other testers. If the numbers differ significantly, have a conversation about why. Maybe you can find new ways to be more efficient or maybe you have some important advice for one of your colleagues.

The team can use the analysis to find bottle necks. E.g. if a lot of time is spent on setup, perhaps the testers need a dedicated test environment that is hooked up to the CI-server or if that is already the case, perhaps it should not re-deploy every time there is a push to the development branch.

So, can metrics improve agile testing?

Yes, if you start measuring right now!

My final recommendation regarding metrics is simply to start collecting data immediately.

Start simple by introducing a simplified version of Session Based Test Management to your testing colleagues on your team – talk to them about it and discuss how your team might benefit. Next, present the idea to the rest of your team and get some feedback on how simple it should be if everyone should be able to use it.

Now, collect some data, analyse it, present the results, and adjust.

Leave a comment about how you use metrics in your organisation and let me know if this helps you in any way.


Here’s a list of the tools you can use to look more into session-based test management an starting a simple measurement of your team’s testing effort:

Rolling box control scheme – The Floor is Lava!

The Floor is Lava / rolling box control scheme
Click the image to try the game.
I prototyped this simple physics based rolling box control scheme using Unity, in only a few hours.
To move the box, you add torque to roll it by pressing the arrow keys or swiping the screen (so far only android devices).
Try to keep the box on the floats and get to the safe platform. If you hit the floor, you’ll lose life.
It feels a bit heavy at first, but it might grow on you with a few other interesting game design dimensions.
Like the controls in Edge, I like the constraints to the three dimensions that it has – not to mention the bonus you’ll get for balancing the box on the edges.
Try out the rolling box control scheme and give me some feedback on it.

QA Specialist in Unity Technologies

For a few months I have been working at Unity Technologies as QA Specialist, handling the publicly submitted bugs reports. I am very proud (even lucky as I have just graduated) of being a part of one of the worlds leading teams in game development.

Our issue tracker, FogBugz, is not feature loaded, but the FogBugz XML API has quite a few possibilities to it.

I am learning Perl to get an easy and versatile tool to work the API with. Once I am on top of that, I will look into how the API can be used to fill in some missing features of FogBugz. Then, in a near future when I have tweaked FogBugz to meet the needs of Unity Technologies, I will put my interests and experience into extending the product itself to make life even more easy for game developers.

Audio and game programmer at DADIU’s Exodroid

While doing my MSc in Medialogy, I took part in two productions at DADIU (Danish Academy for Digital Interactive Entertainment).

My second production was Exodroid and was made in Unity using C#. This production took place in March 2009 while I was writing my master thesis, so these projects are highly interconnected.

As game programmer on this project, I gained experience using C# with Unity. Furthermore, I took an active part in developing the design and the production itself.

I participated in the project while writing my master thesis, therefore much of the discussion and conclusion is somewhat related to this project.