As a tester in an agile software development project, I want to plan and document exploratory tests, so I can prove test results to customers and easily re-run the tests for re-testing and regression testing. This article is made to answer the question of how to plan and document testing in agile projects.
The problems are that time is spent documenting tests that are only used once or that the tests are only run by the same person, so the documentation is never actually read. Other times, the test documentation needs to be handed off to another person that need to run the tests. In the latter situations, the person getting the documentation rarely have time to read the documentation or the required background knowledge to understand it thoroughly without talking to the author anyway.
To solve these problems you should optimise processes within and around your team. Here are some basics that your team need:
- Reliable infrastructure that supports testing
- A shared understanding of basic test techniques
- Thorough user story design with the core team
- Work in pairs to spread knowledge and prevent silos
- Structure manual testing in a simple framework
- Automate regression tests of business critical features
Reliable infrastructure that supports testing
No experienced tester that I have met have not tried to sit and wait for a test environment to be ready or their test data has been corrupted. The problem is that often there are a limited amount of environments or some service or database that the environment depends upon is unstable or not working as expected, so the test cannot be completed.
Building and maintaining a reliable infrastructure for testing can involve many tiny factors that are easy to solve but just as easy to break if unaware of them.
Some of the solutions to these problems are so simple that it is unbelievable that the problems ever existed, but that is easy to say when looking back at them.
Use a dedicated server for testing
In some projects, there are no rule or framework for how servers, database connections, and services are versioned and how they are integrated. Perhaps the team has a few dedicated servers for each phase in the application life cycle
Talk to your team about configuration management and start by naming the environments and make up rules for how they can integrate so no-one accidentally wipes your test environment when you are almost done with the long regression test.
Keep a one-to-one ratio between application and databases
When several application instances are using the same database instance you are begging for trouble. When a change is made to the application in one environment that involves updating the database schema, naturally all of the dependent application instances needs to get that code change as well. The simple solution is to have a one-to-one relation between app-instances and db-instances. Never let more than one code branch use the same database.
Organise test data
When several people are using the same database for testing, say one is automating regression testing and another is running exploratory testing on some new feature, make sure to avoid each other in the database. That means that you need to talk to each other about what data you need to manipulate. If that is unclear or if you are not sure if each other’s data will be affected by proxy, branch out, make a copy of the database and use separate environments. Otherwise you cannot trust your tests anyway.
If part of the requirements are to support loading data from an external service and furthermore serving your data along with that, make sure to validate the integrity of the data from the external source before updating your database with it. If that have already happened, talk to your customer about how to use the external source without breaking your system.
Mock or stub external integrations
Unavailable and unstable integrations in a test environment can be the source of a lot of unwanted noise.
If you are in the unfortunate situation to depend on an external service, make sure to mock it off or virtualise it if you must. That way you have complete control of how it performs. There is not many things worse than being benched because some external resource is performing badly.
A shared understanding of basic test techniques
A common pitfall when available test resources are limited when the deadline approaches, is to get free hands on the team to test without them knowing how to apply test techniques in practise. Some ways to circumvent this is to share knowledge about test techniques within the team, send team members off on courses in practical use of basic test techniques, and/or take on agile coaches or test mentors that can help the team learn some test techniques.
Another idea is to make a book club on the team, where each team member will read a book and present the key points to the rest of the team followed by a discussion that will produce a practical action plan for implementation of at least one idea from the book. Ideally the book club should not be limited to literature about testing, but to anything relevant to the team.
Build a shared reference
Build a library of simplified pattern-descriptions of most test techniques so your team can easily use them in appropriate situations. A place to find relevant information about test techniques is in some of the popular test certification programs, e.g. ISTQB Advanced Test Analyst Syllabus 2012 has a list of some basic techniques that can be used in practise. The following list is taken from the learning objectives list from the chapter on testing techniques. As a tester on your team, you should understand each of them and pick a few that you want to teach your colleagues:
Thorough user story design with the core team
Design user stories together. In software development there are always three very different views on user stories, the customer’s, the coder’s, and the tester’s. An agile practice, called The Three Amigos has been defined in order to describe this fact, how to acknowledge it, and use it as an advantage to design better user stories.
The basics of the practice is that the product owner/business analyst, a coder, and a tester get the joint responsibility to make user stories are ready for implementation and testing according to the customer’s needs. A clever book is written about this: Lean-Agile Acceptance Test-Driven Development: Better Software Through Collaboration
Work in pairs to spread knowledge and prevent silos
Face-to-face interaction is by far the best way to share knowledge. This is even more true when sharing a technique for accomplishing a complex task such as testing an application or teaching someone a business subject matter that might not make sense.
With testing in focus, all team members can benefit from pair-testing. The testers’ mindset can be shared with the other roles on the team so they can build in quality in their respective areas of responsibility on the project. These are some of the benefits that the team roles can draw from pair-testing:
Each tester in the pair can learn details in test techniques from the other tester.
More details are discovered and there are generated ideas on two levels while testing because one tester drives and the other navigates.
If the navigator take session notes, the driver has free hands to follow the flow of digging out a bug from a complex string of actions.
If the driver discovers an unexpected behaviour, the navigator can easier retrace the steps to reproduce the issue.
The tester can pickup som of the business processes that might not make sense if not working with them directly with a product owner.
The product owner can learn some structures and techniques around testing that enables him/her to take over some of the testing tasks when time is limited.
The tester contribute with the tester’s mindset when identifying acceptance criteria to a user story, so these will be more detailed and accurate when the coder starts implementing them and when the tester later need to remember what details matter the most in the story.
The coder can explain some of the logic behind the implementations in the application so the tester can see what areas, modules, or combinations might make more sense to test than others.
The tester can show some of the techniques s/he uses when running a functional test, so the coder can use some of the same techniques when writing unit tests.
The coder can teach the tester about how to structure the architecture behind the automation framework.
Coder/coder, product owner/coder, and product owner/product owner
The last three combinations will also bring a lot of value to the product or project in relation to other tasks than testing.
The coder/coder pairs will produce high-quality code with built-in code review and, if done right, full unit test coverage for regression testing in the CI-system.
The product owner/coder pair will take form as deep discussions into the minute details about the business logic, so the product owner will write more effective acceptance criteria, and the coder will understand more of the undocumented details about the business.
The product owner/product owner pair is rarely done in practice, but the effect of the combination is that the user stories will be more elaborate and accurate.
Structure manual testing in a simple framework
Use light-weight documentation. Once the user stories are ready for implementation and testing according to the customer’s needs som might think that documentation is sufficient. Ideally, yes. In reality, no: details are found during implementation and testing and naturally the decisions around those need to be added to the user story. These might be elaborations around a newly discovered acceptance criteria or an error that occur as a result of the implementation. In relation to testing, add documentation will be descriptions of how various testing techniques would cover the acceptance criteria in the story.
If using an issue tracking system like Jira to track user stories, I recommend simply adding new information to and grooming existing info in the user story until it is done.
Specific to exploratory testing, a useful tool is Session Based Test Management (SBTM) or SBTLite, where the tester will note what the time is spent on while doing exploratory test sessions.
Automate regression tests of business critical features
Regression and re-tests are some of the most time-wasting activities in software development. Mostly because it does not need creative problem solving skills, just mere monkey–see–monkey–do skills. Automate functional tests that you don’t want to repeat ever again.
Think automation from the beginning of the project. Sometimes a considered framework may be very hard to automate. A common problem in UI automation is some UI frameworks’ use of dynamic element identifiers. That makes it hard to write and maintain a good element locator.
Re-testing of all defects that are found
When a defect is found, the test for it will be re-run several times, first by the coder while fixing it, then by the tester that found it. It is a waste of time if the exact same test is run manually again and again. If the tester can automate the re-test of the defect as it is found, it will be a great service to the coder that fixes it. This does not mean that the coder does not have to write unit tests around the fix.
Regression tests that touch high-value business flows in the application
Regression tests are indispensable. That is especially true when a development team is using a VCS with several branches and integrating them often in an agile project. If you manage your automation right, you can make the time spent on manual regression test count. After all, it is with manual testing that the really nasty bugs are found.
Follow the money when writing automated tests. If your customer is always sure to stay in business when you deliver your software, they can pay you. In release-, sprint planning and backlog grooming, there should be made conscious decisions about what is the most important and high value user stories. When automating regression tests, make sure to have a good test coverage of these stories.
Validate integrations that may break during development
You always want to know the state of your own code and that can be delimited by knowing the state of any integrations that affect your system. If an external service is down and one of your features depends on it you have the chance to mock it or virtualise it if it needs more intelligent features than what a mock can provide.
Start criteria for any manual testing should be known
Before giving any human a test related task you should know the basic state of the system. It is very annoying to manually test a long flow only to find out that some depending configuration is not setup correctly and you have to start all over.