7 things to show your skills at the practical CAT exam

At the practical exam for the Certified Agile Tester, there are 7 product types in which you need to show your mastery. Before my CAT exam I took some time to reflect on the essence of each of the products and how I could best show that I know how to test in an agile team. In this article I’ll list my notes on the 7 things to show your skills at the practical CAT exam. I strongly suggest that you take some time to do that as well.

  1. Sprint Plan
  2. Test Strategy
  3. Task Board
  4. Sprint Burn-down Chart
  5. Session Sheets
  6. Sprint Review
  7. Sprint Retrospective

When you’re done reading through my notes, please leave a comment about how you will deliver your favourite .

Because there are seven product types, it doesn’t mean that you’ll pass if you only deliver seven – it is said that it has been done, but whoever made it with only one of each must have been very exact and to the point. Remember that the more content you deliver, the higher your chances are for success because what ever rubbish you deliver just doesn’t count. You can never lose marks only gain them, so train your handwriting and produce as much as you can.

1. Sprint Plan

This is the first thing I did at the exam – even while the instructor and the invigilator where setting up the computers, I was scribbling away just to get it out of my head.

At the practical exam the SUT simulation is delivered in three drops that all need initial testing, regression testing, and eventually some re-testing of the defects that you will find. Apart from that you’ll spend your time on planning, analysing the material, and creating the products that you’ll deliver (i.e. each of these seven products)

It is crucial to set time boxes and to be strict on yourself to uphold them. Start practicing now: how much time will you spend on reading and commenting on this article? I found that my time scheduling slipped when testing the drops, so in retrospective, I’d like to have made smaller time boxes than I initially did (have a look at my initial plan below).

First I thought it was okay to slack a bit on the time boxes in the testing time, but it turned out to be where I needed to be the most strict.

Because my native language is Danish, I had more time for the exam because it was in English, so my sprint plan looked like this:

30 min: Planning
* Test Strategy
* Task board
* Initial Burndown

30 min: Drop 1
* Analysis
* Testing
* Reporting

30 min: Drop 2
* Analysis
* Regression & Re-Testing
* Testing
* Reporting

30 min: Drop 3
* Analysis
* Regression & Re-Testing
* Testing
* Reporting

10 min: Review
* List defects (before the exam I planned to list passed and failed stories, but I ended up listing > those at the Sprint Retrospective)

20 min: Retrospective
* Listed the passed and failed stories. I.e. I was behind my planning so I didn’t get to test the third story properly. I compensated by explaining what parts of it that I did test and that it would go into the next sprint in a real life situation. (Before the exam I really did plan to explain how to mitigate any of the types of failures that I’ve seen in the exam.)
How much detail will you put into your Sprint Plan and how will you practice and enforce the time boxes that you set for the tasks? Share your thoughts in the comments.

2. Test Strategy

Start by making a bare-bone strategy with only the most basic statements that describe how you will test throughout the practical exam.

After each test session, or after you’re done testing a drop, take a minute to update the strategy by adding any methods that you’ve used during the session. Also make sure to take notice of which methods that you added earlier and have not used yet. This will be valuable information for the Sprint Retrospective later on.

These are examples on the basic methods that I wanted to use for my exam. Be sure to use your own favourites, and go through those listed in the manual, then take a look at the stories that you will need to test and add any that will make sense in your specific exam situation.

  • Risk assessment: how to start with user stories that has the highest business value or user stories that have other depending stories (this one didn’t make sense in my situation because each drop had a specific story and the order was predefined with the highest business value first. It would however make a lot of sense to include this statement in a real world test strategy)
  • How tests will be conducted: which test activities, types, techniques, regression testing, re-testing, testing levels, etc.

    • Exploratory testing
    • Regression testing after each new drop
    • Re-testing of defects reported in earlier drops
    • A short description of the use of session sheets for documenting the test sessions (more why than how)
    • How acceptance criteria are used to design tests that the Definition of Done (DoD) include that all acceptance criteria must pass before a user story can be accepted at the sprint review (I really struggled with this at the exam, because all of the stories had some AC that didn’t make it. My conclusion is that the exercise here was to make a DoD that made room for technical and testing debt and to set a threshold for what was acceptable for the product owner)
    • That the tester will inspect both the front-end (UI) and the back-end (database)
    • That any defects will be reported on the session sheets (i.e. not on separate sheets)

Take a look in the manual on what else a test strategy should contain. These notes are reflecting my style and I am looking forward to reading about yours in the comments.

3. Task Board

At the practical exam, there are no real opportunity to use a task board, so you are only asked to make a sketch of one. Make sure to clearly show how it would work in a real situation and include a legend and any additional explanations that you need to clearly explain how your Task Board would be used.

Because it is not actively used, it only eats your time , so you should be very careful not to spend too much time on the sketch.

What does your favourite Task Board look like? In what genius ways have your team’s task board evolved over time?

4. Sprint Burn-down Chart

Compared to the Task Board, the Sprint Burn-down Chart can actually be used, so here are my notes on how I did it at the exam and what I had planned.

In my first attempt at the exam, I found so many severe defects that I didn’t feel I could pass any of them. In hindsight, I should have passed some of the defects on to “the next sprint” as testing debt and passed the (some of the) stories with notes on that in the retrospective. At my second attempt, I only passed two out of three stories and my Burn-down Chart was more interesting to look at and not just a flat-line.

In your planning you will give each of the stories story points on a given scale (in both my cases it was 1 through 6), so you will simply plot the initial sum of the points at drop zero and draw a line for your expected velocity – i.e. when you have estimated the stories and create the Sprint Burn-down Chart. As you finish each drop, plot your progress in the chart.

5. Session Sheets

The Session Sheets are used throughout the practical exam. The difference from the rest of the products is that there is a special way to name your testing sheets so they can be traced to the user stories. Be sure to check the syntax in the practical exercises that you’ll get at the course.

At the exam I also got copies of the user stories with a field for my notes. I was not entirely sure what to do with it, so I simply made some notes related to the acceptance criteria that were listed on the story card.

Because the Session Sheets are the core products of the practical exam, I set up some rules for myself before the exam:

  • Refer directly to Acceptance Criteria and make notes in the actions taken to test them.
  • Not all AC are fulfilled in one code drop, so be specific on which AC are and can be tested in a certain session/drop.
  • Name defects according to AC and any other odd behaviour should be noted as Issues (there are a field for defects and a field for issues below a larger field for session notes on each session sheet)
  • Make a note in the strategy that any AC that are not mentioned is passed. I crossed this out and made a note that the danger of this is that the AC will too easily be forgotten. Instead: Note which AC that are tested and which are left for later drops – even if they are so simple as “The site can be accessed”.
  • When AC pass, create a regression test sheet per story per user story in Drop 2 and 3. On them make a list of the passed AC and check them off as they pass your regression test.
  • If there is a regression, make a specific note on it and note a defect. This actually happened in my second drop. In my first attempt at the exam, I didn’t make any specific regression tests, so I hope this makes a difference.
  • Only create regression test sheets for AC that previously passed and are not implicitly tested with other AC that specifically belong to the current drop. E.g. if an AC says “The site can be accessed”, it will be implicitly be regression tested with any new testing that you do.
    The point of my rules was to be specific about what to make note of and what not to waste time on. In essence, my rules reflect my weaknesses and act as a declaration towards addressing those. Make sure to reflect on your own weaknesses and how you’ll keep them in check during the exam.

In the comments, share the one most important rule that you will set for yourself when taking session notes?

6. Sprint Review

As you can see in my Sprint Plan, the Sprint Review has the smallest time box. All I found necessary was to list the unfixed defects, related to the user stories. Then I added a note simulating a product owner’s decision taken on the customer’s behalf.

Any defects not directly related to acceptance criteria, but related to the additional requirement specifications will also be listed here if they were not fixed. Don’t add any details about the defects. I didn’t even write out the descriptions/titles of the defects, just the IDs that were also listed on the session sheets.

Bear in mind that the Sprint Review should simulate a demonstration of what a team has a accomplished during a sprint, so it is an honest report on what happened and what was delivered. The core value to both the customer and the team, in relation to the collaboration, is the honesty because it makes it clearer what can be expected in the next sprint. The same holds true at the exam where you need to show your professionalism regarding the state of the SUT and that you have done what was in your power, as the tester, to contribute to the product.

As with the other topics, make sure to read the manual and take some time to reflect on how your sprint review will turn out at the exam. If you already have a clear idea, put it in the comments.

7. Sprint Retrospective

The sprint retrospective sheet contain notes about how the process can be improved in the future. In the exam situation, it will should give rational ideas on how your behaviour in the exam process was and how it could be improved.

The retrospective can explain symptoms on what the simulated team can improve. E.g. if there are many security related defects, you can suggest to bring on a security expert to test the product. Likewise with any performance and user experience issues or defects you may find.

If you can see that a majority of the defects could have been prevented with code reviews, unit tests, clearer user stories and acceptance criteria, the Sprint Retrospective is where you should address those.

Most important is to address all the issues that are noted on the session sheets during the tests – and make sure to make notes on any issues like these, on the session sheets, when you find the issues.

Explain which issues are most important to the team and the customer and how the team would mitigate these.

Issues are e.g. if stories are too big, label them as epics and break them down into smaller stories. If acceptance criteria are unclear, explain that the testers should pair up with the product owner to make them testable.

If you have an additional example, post it in the comments and explain what a symptom could be and how the issue can be a problem in the future.

CAT Exam

In this article I’ll give you an idea of how my exam for the Certified Agile Tester (CAT) went, how I prepared, if I was nervous, and if I thought it was hard.

How did I do?

The first time I took the exam I didn’t pass. With the risk of sounding like an excuse, I’ll try to explain why I thought I failed the first attempt. Perhaps my reflections will help some of you pass the exam the first time. The iSQI does not provide any other information than whether you pass or fail the exam and the percentage of correct answers in each test (the practical exam and the written exam).

Reasons why I may have failed my first attempt at the CAT Exam:

  • Read too little
  • Wrote too little answers (focused on being right in every answer)
  • Had higher thoughts of my knowledge of agile testing than anticipated
  • Practical
    • Missed “required” sheets
    • Unclear traceability between strategy, plan, tests, defects, and re-tests
    • Too few defects reported
  • Written
    • Need more corrects answers (remember that no incorrect answers can subtract from your score, only correct answers can add to your score)
    • Practice more with the sample questions in the CAT Manual
    • Need to write faster to give more answers

In the time leading up to my second attempt, I worked on these issues and it seemed to have worked as I passed the exam with much better scores than the first time. You may need to work on the same things, but try to think about what issues you may run into and work on those before you may have to compile a list like this after a first attempt.

How did I prepare for the exam?

I have always hated preparing for exams because there is so much pointless tension connected to it. It is really stressing me out. On the other hand I really love researching a topic for use in a presentation, an article, or similar. Given I had a bit extra time in the weeks before the course, I decided to create a website about the CAT course, to help others study for the CAT exam. With that decision I added a real purpose to my effort and removed the pointlessness of the tension.

Was I categorising too much?

In the beginning I spent a lot of time grasping the structure of the CAT Manual 3.0. I listed all the learning objectives for each module with their K-levels. I phrased the learning objectives as titles on articles that would become part of the website. For each of the articles/learning objectives, I listed a handful of questions that would answer the key topics of the learning objective. The idea was then to simply fill out the questions, supplement with what would learn from the CAT Manual, and later from the CAT Course itself.

In hindsight, this was a big mistake and it is completely against the Agile manifesto and principles of just-in-time learning. Even though I wanted to get an overview of the CAT Manual to be able to start with what I would find the most important or the hardest to learn, I ended up wasting time on categorisation, prioritisation, and the whole idea of how the website would be like. The CAT Manual is carefully designed for the students to follow from start to finish and I should simply have done so, writing my notes to my articles as I went through it before the course.

As I am writing this, I worry that I am making the same mistake again, by spending time on reflecting on the exam outcome. However, this will be my only opportunity to do so. The next steps after this reflection is to read the CAT Manual again and fill in my notes in the article scaffolds that I already created. While waiting for the exam results, I wanted to use the opportunity of having them fresh in memory, to summarise my experiences of each day on the course. I didn’t expect that I would have to take the exam again.

Was the exam hard?

Before giving my subjective opinion on it, I will try to retell some of the other participants’ concerns especially on the practical exam, which we discussed in the lunch break.

There is a lot of handwriting involved in the exam. As I suspect most people in IT, I am not practicing pen writing at the level required by the CAT exam. Of course at the written exam, there is a lot of writing to do, but there it is really simple and you only have to concentrate on one thing: answering the question within the given time. At the practical exam, you need to write a lot as well, only using another technique. You also have to juggle a lot of things and be careful to make all your notes traceable all the way from the bugs that you find and back to the specifications and the user stories. On top of that, there are the required products that you need to deliver.

At my first attempt at the exam, one of my colleagues had some technical issues with the linux distribution that was running the assignment. I really felt for him while I was glad that it wasn’t happening to me. I’m not sure I could have handled the added stress very well. Believe it or not, at my second attempt I found myself in a similar situation, where I wasn’t provided with the credentials to the database. In both instances, it was handled very well by the instructor and the invigilators and both my colleague and I were given extra time as compensation.

The written exam questions were very much like the sample questions. At least, I did not find them hard to understand at all. That said, I should probably have put more effort into answering them with more variations – I only answered with the required number of answers and no more.

Why is the exam so long?

There is a lot to cover, so it probably cannot be any shorter. Many topics are covered in the CAT Manual and the CAT Course, so to test the participants in as many of them as possible, the exam has to be long.

Was I nervous about the exam?

I wasn’t really nervous because I was confident that I would pass the exam. I based my confidence on the fact that I work by the principles presented in the CAT Manual and that my first encounter with software testing theory was a lot of self studying and reading. I felt that I had seen and heard of everything in the CAT Manual and had tried most of it. To say the least, I was very confident that I knew the subject matter.

How was my effort before and during the exam?

Perhaps I lost track of the primary goal, of passing the exam, when I shifted focus from only studying for the exam to using what I learn to building a website to help others take the exam. My first impression of my effort is that I spent a lot of time and energy on researching and studying the CAT Manual, but thinking back on it, I spent too much of my time and energy grasping the structure of the CAT Manual instead of memorising the actual content.

Leading up to the exam I let myself be tricked into thinking that the Lean principles of delivering just enough to give value also applies to the exam situation. That is NOT the case here! I would answer each question in the written exam with just enough information to satisfy myself. I did not give any variations on the answers, making my answers more clear and increasing the chance of gaining extra points.