How to deliver test statuses at a daily scrum

As an agile tester, I want to share the status of the project’s testing tasks with my team, so we can solve any issues and impediments before they become bottle necks.

This article will give you an idea of how to deliver test statuses at a daily scrum. The test related work in agile teams seems to revolve around user stories and their acceptance criteria. This will naturally also be the centre of the discussions in daily standup meetings (or a scrum as it is often called).

Here are some of the basic question that I will address in this article:

  • What is the basic agenda of a stand-up meeting?
  • Which testing activities should an agile tester be planning for an iteration?
  • What testing constraints should an agile tester be prepared to meet?
  • What obstacles impacting testing should an agile tester be prepared to address?
  • What to consider when communicating in cross-functional teams
  • The beauty of Failing Fast and sharing it based on trust and shared responsibility

How To Deliver Test Statuses at Daily Scrums so Everyone can Understand It

One of the rules of the scrum is that it is time-boxed, so you don’t get to chit-chat – you have to be concise. Another scrum rule is that each and all members deliver a status including obstacles and next steps within the time-box. It is pretty much like a turn-based game where each player go through the turn’s phases:

  1. what was your tasks since last scrum
  2. what are your tasks until next scrum
  3. what may prevent you from accomplishing them

A case: You’ve been working with a coder on a user story for an email opt-in form on a website:

“As a non-subscriber,
I can submit my email on a landing page,
so I can receive updates through the email marketing channel”

You’ve established the acceptance criteria and the need to do an exploratory test that include an integration to an email marketing provider, but you don’t have the customer’s credentials for the service so you can setup a test environment for the integration. You’ve already collaborated with the coder on the tests and there is a stub that can receive the email.

This is how you’d give your team the status update:

“Yesterday, Coder Bob an I tested the story for the email opt-in form on the landing page with a stub.
Next step is to test that the integration to the email marketing provider actually works.
However, I need the credentials to the email marketing service so I can setup an email-list that can catch the output from the form”

To that, the product owner would naturally answer:

“Sure, I’ll get it for you after the meeting”

There: three breaths of air and everyone on the team knows exactly what you did, what you’re about to, and what holds you back from doing it. You even got a solution to your problem in less than 30 seconds. (I should add that the example story is pretty short and there would be more of the same kind in your status update. Problems of this size could also be solved outside the standup meeting, but for the sake of the example and me not being able to come up with a real problem, please play along here).

In the case, you clearly state your task since last stand-up meeting, your next step, and the obstacle you’re facing. A team member have the solution to your problem and it is clear what has to be done and who will take the next step.

However, what if not all team members know what you’re talking about, even if you speak no riddles?

In agile teams all members have different skill sets, which is good. It is therefore important, not only to be concise but to also provide enough information about an obstacle so everyone can understand it and potentially help you clear the obstacle.

It is a two-way street: if you find yourself in a scrum not understanding a fellow team member’s problems it is your job to ask for just enough details to understand the problem. Don’t babble on or encourage the speaker to do so. Simply raise your hand and ask concisely. To riff on the example, you might not know what an email-list is in the context of email marketing providers. Ask: “What is an email-list”. Everyone at the scrum will know the context of the question and know that you ask because you might have a great solution to the problem if you know more about it.

Fear of inadequacy might withhold some people from asking that “dumb” question. Fear of inadequacy and “dumb” questions simply does not exist in pure collaborative work. You’re already a team member, so you’ve probably been chosen for a unique skill set that fill the gaps of other team members – perhaps even your ability to ask questions in a way that reveal unforeseen issues, in which case it is your duty to practise that skill until success.

  • Be concise.
  • State what you did,
  • what you’re about to,
  • and what may hold you back.
  • If in doubt, ask
  • and ask again.

What is your take on how to deliver test statuses at a daily scrum? Do you share similar experiences or is there a side to it that you think is important? Leave a comment and let me know.

Can metrics improve agile testing? Measure, Analyse, Adjust

As a tester in an agile project, I want to make visible how I spend my time, so my team can act accordingly and stakeholders know what to expect.

As you’ve probably experienced, plans and estimates are revised during a sprint. I have never heard of any healthy project that has not done that. In an unhealthy project, stakeholders may stick to a loosely estimated initial plan because they do not know better and cannot act on any historic data

Consider a scenario with a release date getting closer. A stakeholder, who is restrained by a contractual budget and a promise of timely releases, asks the team when they are done testing the stories. The team can’t answer with any acceptable margin, so the stakeholder start drilling down into the stories asking specific questions about how many hours are needed for exploratory testing, test design, and GUI automation – basically taking over the test strategy. At the end of the meeting the testers have committed to some arbitrary forced number of hours for each type of testing on the remaining user stories. The only benefit of that is that the stakeholder is satisfied until it is discovered that none of the new estimates are useful at the next status meeting.

In this article I provide an answer to the question: Can metrics improve agile testing. The best way to answer that is to split it into these four sub topics which I will answer in sequence:

  • Why collecting metrics is helpful
  • What metrics should be gathered during a sprint?
  • What is the easiest way to gather metrics during a sprint?
  • How can metrics help improve testing?

Everyone must know why they are collecting metrics

Imagine recording data without knowing what it is used for and why it is gathered in the first place. The first thing I could imagine myself doing was – consciously or unconsciously – making up reasons. Adding to that, I would become insecure if the metrics are in any way connected to my performance on the team. Building on that and the uncertain reasons behind the metrics, some might even alter those in a direction that would be of no benefit to anyone.

Say, for example, that you are recording how much time you spend on testing a story. You have no idea what the data is used for. You want yourself and the project to look good in the statistics, so you start rounding the numbers down here and there – just a tad. Over the duration of the sprint, that become several hours. Then it is time for stacking the backlog for the next sprint and guess what more stories are taken in than you and your team can test within the sprint.

So be transparent about why and how metrics are gathered. Be vocal about why you measure any performance on the team and project. If you are in any doubt of why you are collecting a metric, ask about it and discuss what it can be used for and how it can benefit the project.

What metrics should be gathered during a sprint?

In other words: What does testers spend their time on? The answer is in my experience always some version of these three things:

  • Testing (including running tests against the application under test)
  • Investigating, reporting, and re-testing bugs (including collaborating with coders and product owners)
  • Setting up and waiting for environments or other resources such as test data (including designing scripted test cases based on notes from previous ET-sessions)

One of the most efficient test techniques in agile testing is exploratory testing because there is less preparation and setup required compared to scripted tests. It is not always easy to gather metrics while focussing your energy on finding creative ways to test the system, so you will need a simple system for doing so.

What is the easiest way to gather metrics during a sprint?

One such system I’ve been using for gathering metrics during exploratory testing sessions is a light version of session-based test management or SBTM, which was developed by Jon and James Bach. The core idea is simply to take notes of what you are doing during a time boxed session and mark each entry. The notes have to really simple so it does not take up all your creative testing energy.

The wikipedia entry defines session based testing like so:

Session-based testing is a software test method that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. wikipedia.org/wiki/Session-based_testing

A genius of the tool is that you will take specific note of how much time you use on
1) Testing, 2) investigating Bugs, and 3) Set up. These are also called TBS Metrics. When you gather the metrics, you can see where you spend your time, so if you see less time testing than investigating bugs or even setting up the environment, you can take action on for example getting the environment setup automated. If e.g. you spend a lot of time investigating bugs, perhaps the programmers should be more involved in testing or the user stories should be analysed with the product owner to get more specific acceptance criteria.

The important point is that if you collect these 3 simple TBS metrics, you have come a long way towards being able to analyse your results, you can continuously improve your testing effort based on those numbers.

The full SBTM may be a big bite to start off with, so I recommend that you start with SBT Lite which was defined by my mentor Sam Kalman who worked with Jon Bach at Quardev. SBT Lite is much more free-form, but is still able capture the metrics you need.

In practise, a tester would take notes on his/her actions during an exploratory test session.
Before the session starts, a session charter is chosen and noted on the session sheet along with the esters name. The charter could be one of the acceptance criteria in a user story or an entire user story depending on the size and scope of them. A meta session charter could also be to come up with session charters for a set of stories. The charter is basically the mission of the session.

During the session, the tester marks his/her notes as T, B, or S, along with a time-stamp and a one-line description of what happened. The result should look much like a commit log from healthy pair programming session.

When analysing the session logs, it would be evident how much time is used on testing, bug, or setup time respectively. In the full SBTM resource there is a link to Perl tool that can analyse session notes if they are formatted in a specific way, but you should make your own session log parser if you prefer to follow the simpler version or even design your own set of metrics (which I recommend once you’ve tried this for a few sessions).

How can metrics help improve testing?

As I suggested in the scenario in the introduction, it is not only managers and stakeholders that benefit from the collected metrics. Testers can benefit on a personal (professional) level, the team can benefit, if there is a cross-team test organisation that too can benefit from the TBS metrics.

The individual can use the analysis of the metrics to hone his/her skills. Perhaps compare how much time is spent on each task type compared to other testers. If the numbers differ significantly, have a conversation about why. Maybe you can find new ways to be more efficient or maybe you have some important advice for one of your colleagues.

The team can use the analysis to find bottle necks. E.g. if a lot of time is spent on setup, perhaps the testers need a dedicated test environment that is hooked up to the CI-server or if that is already the case, perhaps it should not re-deploy every time there is a push to the development branch.

So, can metrics improve agile testing?

Yes, if you start measuring right now!

My final recommendation regarding metrics is simply to start collecting data immediately.

Start simple by introducing a simplified version of Session Based Test Management to your testing colleagues on your team – talk to them about it and discuss how your team might benefit. Next, present the idea to the rest of your team and get some feedback on how simple it should be if everyone should be able to use it.

Now, collect some data, analyse it, present the results, and adjust.

Leave a comment about how you use metrics in your organisation and let me know if this helps you in any way.

Resources

Here’s a list of the tools you can use to look more into session-based test management an starting a simple measurement of your team’s testing effort:

How to Plan and Document Testing in Agile Projects

As a tester in an agile software development project, I want to plan and document exploratory tests, so I can prove test results to customers and easily re-run the tests for re-testing and regression testing. This article is made to answer the question of how to plan and document testing in agile projects.

The problems are that time is spent documenting tests that are only used once or that the tests are only run by the same person, so the documentation is never actually read. Other times, the test documentation needs to be handed off to another person that need to run the tests. In the latter situations, the person getting the documentation rarely have time to read the documentation or the required background knowledge to understand it thoroughly without talking to the author anyway.

To solve these problems you should optimise processes within and around your team. Here are some basics that your team need:

  • Reliable infrastructure that supports testing
  • A shared understanding of basic test techniques
  • Thorough user story design with the core team
  • Work in pairs to spread knowledge and prevent silos
  • Structure manual testing in a simple framework
  • Automate regression tests of business critical features

Reliable infrastructure that supports testing

No experienced tester that I have met have not tried to sit and wait for a test environment to be ready or their test data has been corrupted. The problem is that often there are a limited amount of environments or some service or database that the environment depends upon is unstable or not working as expected, so the test cannot be completed.

Building and maintaining a reliable infrastructure for testing can involve many tiny factors that are easy to solve but just as easy to break if unaware of them.

Some of the solutions to these problems are so simple that it is unbelievable that the problems ever existed, but that is easy to say when looking back at them.

Use a dedicated server for testing

In some projects, there are no rule or framework for how servers, database connections, and services are versioned and how they are integrated. Perhaps the team has a few dedicated servers for each phase in the application life cycle

Talk to your team about configuration management and start by naming the environments and make up rules for how they can integrate so no-one accidentally wipes your test environment when you are almost done with the long regression test.

Keep a one-to-one ratio between application and databases

When several application instances are using the same database instance you are begging for trouble. When a change is made to the application in one environment that involves updating the database schema, naturally all of the dependent application instances needs to get that code change as well. The simple solution is to have a one-to-one relation between app-instances and db-instances. Never let more than one code branch use the same database.

Organise test data

When several people are using the same database for testing, say one is automating regression testing and another is running exploratory testing on some new feature, make sure to avoid each other in the database. That means that you need to talk to each other about what data you need to manipulate. If that is unclear or if you are not sure if each other’s data will be affected by proxy, branch out, make a copy of the database and use separate environments. Otherwise you cannot trust your tests anyway.

If part of the requirements are to support loading data from an external service and furthermore serving your data along with that, make sure to validate the integrity of the data from the external source before updating your database with it. If that have already happened, talk to your customer about how to use the external source without breaking your system.

Mock or stub external integrations

Unavailable and unstable integrations in a test environment can be the source of a lot of unwanted noise.

If you are in the unfortunate situation to depend on an external service, make sure to mock it off or virtualise it if you must. That way you have complete control of how it performs. There is not many things worse than being benched because some external resource is performing badly.

A shared understanding of basic test techniques

A common pitfall when available test resources are limited when the deadline approaches, is to get free hands on the team to test without them knowing how to apply test techniques in practise. Some ways to circumvent this is to share knowledge about test techniques within the team, send team members off on courses in practical use of basic test techniques, and/or take on agile coaches or test mentors that can help the team learn some test techniques.

Another idea is to make a book club on the team, where each team member will read a book and present the key points to the rest of the team followed by a discussion that will produce a practical action plan for implementation of at least one idea from the book. Ideally the book club should not be limited to literature about testing, but to anything relevant to the team.

Build a shared reference

Build a library of simplified pattern-descriptions of most test techniques so your team can easily use them in appropriate situations. A place to find relevant information about test techniques is in some of the popular test certification programs, e.g. ISTQB Advanced Test Analyst Syllabus 2012 has a list of some basic techniques that can be used in practise. The following list is taken from the learning objectives list from the chapter on testing techniques. As a tester on your team, you should understand each of them and pick a few that you want to teach your colleagues:

Specification-Based Techniques

Defect-Based Techniques

Experience-Based Techniques

Thorough user story design with the core team

Design user stories together. In software development there are always three very different views on user stories, the customer’s, the coder’s, and the tester’s. An agile practice, called The Three Amigos has been defined in order to describe this fact, how to acknowledge it, and use it as an advantage to design better user stories.

The basics of the practice is that the product owner/business analyst, a coder, and a tester get the joint responsibility to make user stories are ready for implementation and testing according to the customer’s needs. A clever book is written about this: Lean-Agile Acceptance Test-Driven Development: Better Software Through Collaboration

Work in pairs to spread knowledge and prevent silos

Face-to-face interaction is by far the best way to share knowledge. This is even more true when sharing a technique for accomplishing a complex task such as testing an application or teaching someone a business subject matter that might not make sense.

With testing in focus, all team members can benefit from pair-testing. The testers’ mindset can be shared with the other roles on the team so they can build in quality in their respective areas of responsibility on the project. These are some of the benefits that the team roles can draw from pair-testing:

Tester/tester

Each tester in the pair can learn details in test techniques from the other tester.

More details are discovered and there are generated ideas on two levels while testing because one tester drives and the other navigates.

If the navigator take session notes, the driver has free hands to follow the flow of digging out a bug from a complex string of actions.

If the driver discovers an unexpected behaviour, the navigator can easier retrace the steps to reproduce the issue.

Product owner/tester

The tester can pickup som of the business processes that might not make sense if not working with them directly with a product owner.

The product owner can learn some structures and techniques around testing that enables him/her to take over some of the testing tasks when time is limited.

The tester contribute with the tester’s mindset when identifying acceptance criteria to a user story, so these will be more detailed and accurate when the coder starts implementing them and when the tester later need to remember what details matter the most in the story.

Coder/tester

The coder can explain some of the logic behind the implementations in the application so the tester can see what areas, modules, or combinations might make more sense to test than others.

The tester can show some of the techniques s/he uses when running a functional test, so the coder can use some of the same techniques when writing unit tests.

The coder can teach the tester about how to structure the architecture behind the automation framework.

Coder/coder, product owner/coder, and product owner/product owner

The last three combinations will also bring a lot of value to the product or project in relation to other tasks than testing.

The coder/coder pairs will produce high-quality code with built-in code review and, if done right, full unit test coverage for regression testing in the CI-system.

The product owner/coder pair will take form as deep discussions into the minute details about the business logic, so the product owner will write more effective acceptance criteria, and the coder will understand more of the undocumented details about the business.

The product owner/product owner pair is rarely done in practice, but the effect of the combination is that the user stories will be more elaborate and accurate.

Structure manual testing in a simple framework

Use light-weight documentation. Once the user stories are ready for implementation and testing according to the customer’s needs som might think that documentation is sufficient. Ideally, yes. In reality, no: details are found during implementation and testing and naturally the decisions around those need to be added to the user story. These might be elaborations around a newly discovered acceptance criteria or an error that occur as a result of the implementation. In relation to testing, add documentation will be descriptions of how various testing techniques would cover the acceptance criteria in the story.

If using an issue tracking system like Jira to track user stories, I recommend simply adding new information to and grooming existing info in the user story until it is done.

Specific to exploratory testing, a useful tool is Session Based Test Management (SBTM) or SBTLite, where the tester will note what the time is spent on while doing exploratory test sessions.

Automate regression tests of business critical features

Regression and re-tests are some of the most time-wasting activities in software development. Mostly because it does not need creative problem solving skills, just mere monkey–see–monkey–do skills. Automate functional tests that you don’t want to repeat ever again.

Think automation from the beginning of the project. Sometimes a considered framework may be very hard to automate. A common problem in UI automation is some UI frameworks’ use of dynamic element identifiers. That makes it hard to write and maintain a good element locator.

Re-testing of all defects that are found

When a defect is found, the test for it will be re-run several times, first by the coder while fixing it, then by the tester that found it. It is a waste of time if the exact same test is run manually again and again. If the tester can automate the re-test of the defect as it is found, it will be a great service to the coder that fixes it. This does not mean that the coder does not have to write unit tests around the fix.

Regression tests that touch high-value business flows in the application

Regression tests are indispensable. That is especially true when a development team is using a VCS with several branches and integrating them often in an agile project. If you manage your automation right, you can make the time spent on manual regression test count. After all, it is with manual testing that the really nasty bugs are found.

Follow the money when writing automated tests. If your customer is always sure to stay in business when you deliver your software, they can pay you. In release-, sprint planning and backlog grooming, there should be made conscious decisions about what is the most important and high value user stories. When automating regression tests, make sure to have a good test coverage of these stories.

Validate integrations that may break during development

You always want to know the state of your own code and that can be delimited by knowing the state of any integrations that affect your system. If an external service is down and one of your features depends on it you have the chance to mock it or virtualise it if it needs more intelligent features than what a mock can provide.

Start criteria for any manual testing should be known

Before giving any human a test related task you should know the basic state of the system. It is very annoying to manually test a long flow only to find out that some depending configuration is not setup correctly and you have to start all over.

Simple Game Design by Polish, Appeal and Measures

The cycles to achieve simple game design
The cycles to achieve simple game design

If you would like to read my master thesis, go ahead and get it here. Otherwise, read on and get an overview of my thoughts behind Simple Game Design through Polish, Appeal & Measures.

With a vantage point in the term Game Polish, I have proposed a simple method for designing games independently from any specific production methodologies.

The study was made in the spring of 2009 through correspondences with members of the International IGDA Game Design SIG and other forums dedicated to game development.

The main cornerstones of the study is Polish, Appeal, and Measures – all elaborated in the report.

Simple Game Design through Polish, Appeal & Measures. This method proposed in my master thesis, explain how to make a game or on what to focus designing it.

An Idea in a Master Thesis

In this thesis the reader will be presented with a definition of the nebulous term Polish used in relation to computer games, mainly derived from literary sources. The term has been processed from different perspectives and split into three overall topics: Polish, Appeal, and Measures.

The topic of Polish relate to the elements used to add Appeal to a computer game. The topic of Measures relate to how the intentions of the game’s Appeal are being verified through the pre–production and the production phases in the making of a computer game. These three topics will form the cornerstones of this master thesis.

A Simple Game Design Method

The overall methodology of the study consist of two parts. The first part being a series of correspondences with game designers via e–mail and online discussion forums dedicated to game design issues. The second phase of the methodology has been to verify the responses through a series of interviews with professional game developers working at different levels in the game industry.

The results are presented according to the cornerstones, Polish, Appeal and Measures reflecting the respondents’ opinions on how the elements of Polish are being evaluated in pre–production and production of computer games. It is clear that players are not included in pre–productions as much as in productions. The elements of Polish should not be treated as those with functional and pleasing purposes separately, as there are rarely added any elements of Polish that does not have a distinct function in the game.

Several aspects of game design are being discussed, however an effort is made towards making the proposed methodology independent of the established definition of staged software development. In relation to this there is being proposed a more fluid way to describe the process of game development through the terms Concept, Design, and Content.

The conclusion of the thesis proposes a simple methodology of designing games that is independent of software production methodologies.

Audio and game programmer at DADIU’s Exodroid

While doing my MSc in Medialogy, I took part in two productions at DADIU (Danish Academy for Digital Interactive Entertainment).

My second production was Exodroid and was made in Unity using C#. This production took place in March 2009 while I was writing my master thesis, so these projects are highly interconnected.

As game programmer on this project, I gained experience using C# with Unity. Furthermore, I took an active part in developing the design and the production itself.

I participated in the project while writing my master thesis, therefore much of the discussion and conclusion is somewhat related to this project.

Using Game Design Patterns in Casual Online Game Productions

9th Semester/Internship

Spending time in “the real world” was a great lesson for me. Besides the academic work, described below, I gained first hand experience in what it takes to deliver production quality flash games. Through this I tried using game design patterns in casual online game productions.

Games should be fun and plenty. If game developers should make plenty of fun games they need efficient tools to design them. Game designers today use existing games as references when discussing game design. Pattern languages may offer a common reference frame and Game Design Patterns may help game developers design more efficiently.

This case study was made to find what it would be like using game design patterns in casual online game productions in a small casual game company. Among the findings was a positive attitude towards the methods, however if a company is to benefit from them there has to be taken measures towards educating staff and redesigning production models.

Whether Game Design Patterns are in fact useful remains uncertain. This study offer suggestions to what is necessary to succeed in implementation of Game Design Patterns based on Björk’s and Holopainen’s research.

DADIU May 2007 – Hængerøv

Hængerøv

My first DADIU production was called Hængerøv and was made in Source SDK using Lua, C++ and Visual Studio 2005. The production took place May 2007.Hængerøv poster

The game was a 3D platformer where you would play a young boy, that accidentally broke his sister’s new cell phone while secretly tampering with it in his tree house one night. The objective is to collect the missing buttons while evading the furious sister who threatens to embarrass you by showing a picture of your bare bottom to the entire school, but that is not all.

The dark garden is a treacherous place to sneak around. Garden gnomes have come to life and they yodel awfully loud when tripped over, and that is what your sister is waiting for, so she can find you and take your picture.

Your parents aren’t of much help as they are having a garden party. Empty wine bottles can also give up your position, and so can the (not-so-scary-almost-pathetic) ghosts that are haunting the garden.

Fortunately you are armed with your Bug Vacuum Gun, that can suck up the ghost. The downside of it is that it attracts your sister’s attention as well.

The only chance of avoiding total embarrassment is to return the fixed mobile to your sister before her boyfriend calls her.