My Intent
I have been asked by a friend and colleague to help her define the transformation plan for the tester’s team in a software development organization. The goal is to help testers learn agile testing techniques, so they can better help the agile development teams they belong to. As generalizing specialists, the organization wants them to be the agile testing experts within their teams.
I am by no means an agile testing guru, even though I have done my fare share of testing in agile teams, but I have seen a few repeating scenarios in the teams I have coached and it is my purpose to document them in this post, as it can be useful to somebody in a similar situation to my friend and in the hope of receiving useful feedback.
Structure of this Post
- Starting Points a description of different roles and stereotypes frequently found in development teams. It will help us understand where we are, that is the beginning of the trip, and what the next steps could be.
- An Agile Tester… tries to define what characterizes an agile tester and provide some guidance on the destination of this trip.
- Agile Testing BOK is a simple list with short descriptions of the different disciplines included in agile testing, structured around the world-famous _agile testing quadrants_.
- Some Frequent Journeys describes some examples of trips to agile testing mastery, starting from a few different realities.
- Learning Resources compiles links to books and articles that can help understanding agile testing and its practices.
Starting Points
It all starts where you are now. My intention in this section is to acknowledge there are as many different starting points as people willing to start a journey. Even seeing such wild diversity I think we can still recognize some common patterns that can be useful as you or the people you are trying to help finding their path may be close to what described herein.
Traditional Senior Tester |
|
Traditional Junior Tester |
|
Traditional Programmer |
|
Traditional Architect / Senior Developer |
|
Business Analyst |
|
Traditional Product Manager |
|
Agile Coder Mid-journey |
|
An Agile Tester…
- focuses on delivering business value.
- is an Agile team member and knows and has applied Agile Software Development basics:
- Manifesto
- Principles
- A simple framework as Scrum and / or Kanban
- knows rudiments of XP Technical Practices and the synergy / relationships between them.
- practices Agile testing, i.e. ATDD, Specification by example, BDD.
- leverages automation as much as it’s practical.
- is flexible, takes a whole team approach to software development.
- is a generalizing specialist so she practices or learns from:
- PO and BA. What the customers need. Domain knowledge. How to write good specifications using tests.
- Developers. TDD, white box testing, architecture and design rudiments and some coding techniques.
- Other Quality experts about non-testing aspects of software quality.
- teaches others in the team about Agile testing (their specialty) in general.
Agile Testing BOK
This section is based on the Agile Testing Quadrants first introduced by Brian Marick here http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2 and later refined by Lisa Crispin here http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants and in her Agile Testing book http://techbus.safaribooksonline.com/book/software-engineering-and-development/software-testing/9780321616944.
If you are not yet familiar with Lisa’s book, do yourself a favor and read it now, it’s a great use of your time even if you are already in your agile testing trip.
Another great resource to acquire insights that can change your view of testing in a record time is Scott Ambler’s magnificent article Agile Testing and Quality Strategies: Discipline Over Rhetoric http://www.ambysoft.com/essays/agileTesting.html. Go read it, now… really!
This is my take on the agile testing quadrants, it contains a combination of ideas obtained from the aforementioned sources plus my own material (of course all errors are only my own).
Q1: Technology-Facing Tests that Support the Team
Purpose | Go faster, do moreCreate flexible code that adapts gracefully Shortest feedback cycle |
Types | Automated Unit Tests Automated Component Tests |
Toolset | xUnit frameworksmockobjects build automation tools source code control |
Audience | developers: create, use and maintain test suites test experts: assist in defining functional content and data sets to be used |
Risks / difficulties | technical debt: legacy code with low (even zero) unit test density complex / heavy / slow dependencies (i.e. databases) Execution of test suite takes longer than acceptable to the team so it’s not run often. |
Frequent Practices | TDDObject Mocking Design for testability |
Q2: Business-Facing Tests that Support the Team
Purpose | These tests help provide the big picture and enough details to guide coding Clarify and specify requirements Risk identification / mitigation in obscure / risky areas of the product Detail conditions of satisfaction Increase the domain knowledge of the team Functional regression safety net |
Types | Automated Acceptance Tests Story Tests Functional Tests Simulations |
Manual | Prototypes Examples Simulations |
Toolset | Eliciting Requirements – Flow-diagrams – Mock-ups (i.e. paper prototypes) – Wireframes Testing behind the GUI – xUNit frameworks – BDD tools (Cuke4Duke, Cucumber-JVM, Concordion, easyb and JBehave for Java, NBehave and NSpec for .NET, and Cucumber and RSpec for Ruby) – FIT / FitNesse / SLiM – CrossCheck, Ruby Test::Unit, soapUI for testing Web Services Testing through the GUI – Record and playback tools – Scripting tools * Watir for Ruby * Selenium * Canoo WebTest Test Management Tools – Geminy – HP Quality Centre – IBM Rational Quality Manager – TestLink – Wiki – FitNesse |
Audience | Test experts * Define together with BAs the tests * Implement automated tests together with developers BAs * Define together with Test-experts the tests Developers * Support the test automation effort * Consume tests as specification and verification tools |
Risks / difficulties | Tests take long to be specified during the sprint. Build tests incrementally to feed the development team high-level acceptance tests early on. Tests are not maintained after the sprint in which they were introduced. To avoid it make sure all tests always pass in your system. Execution of test suite takes longer than acceptable to the team so it’s not run often. technical debt: legacy code with low (even zero) unit test density complex / heavy / slow dependencies (i.e. databases) |
Frequent Practices | BDD ATDD Specification-by-example Data-driven testing Testing-behind-the-UI Testing-through-the-UI |
Q3: Business-Facing Tests that Critique the Product
Purpose | |
Types | Exploratory TestingScenariosUsability Testing
UAT (User Acceptance Tests) Alpha / Beta Usage |
Toolset | Test Setup Automation using any of the tools mentioned in Q2 and Q1Test Data Generation tools (i.e. PerlClip)Log File Monitoring Tools (i.e. Unix’s tail, LogWatch)
Simulators Emulators Scenarios and Workflows |
Audience | Test-experts
BAs
Technical-writers
Developers
|
Risks / difficulties | Difficult to automate as it needs “a brain” to critique the product.If Q1 and Q2 tests don’t leverage automation properly there will be no time for Q3 testing.Sometimes it’s difficult to engage the relevant stakeholders into this kind of testing, especially when done in a rolling-wave iterative approach as some of them are accustomed to end-of-project testing stage. |
Frequent Practices | DemonstrationsEnd-to-end systemTest-behind-the-UI
Soap Opera Testing, a term coined by Hans Buwalda [2003]: Take a scenario that is based on real life, exaggerate it in a manner similar to the way TV soap operas exaggerate behavior and emotions, and compress it into a quick sequence of events. Think about questions like, “What’s the worst thing that can happen, and how did it happen?” Use Automation to help in exploratory testing (i.e. setup, frequently performed sequences). User Needs and Persona Testing for Usability testing |
Q4: Technology-Facing Tests that Critique the Product
Purpose | Verify nonfunctional requirements including configuration issues, security, performance, memory management, the “ilities” (e.g., reliability, interoperability, and scalability), recovery, and even data conversion. |
Types | Performance and Load TestingStress Testing”ilities” testing
|
Toolset | Performance and Load Testing ToolsUnitl Level Perf Tools: JUnitPerf, httperfOpen Source: Apache JMeter, The Grinder, Pounder, ftptt, and OpenWebLoad
Commercial: NeoLoad, WebLoad, eValid LoadTest, LoadRunner, and SOATest. OS Profiling Tools Ethical Hacking Tools |
Audience | Developers
Test-experts
Security expert (Ethical Hacker)
Database Experts
Performance Test Expert
|
Risks / difficulties | The team may get focused on the business requirements and forget about the non-functional ones. This may be even perceived as something to be dealt with by developers only.When they are perceived as low-risk for a project they may be missing completely from the test plan.They need special knowledge and expensive tools to be performed so they are “faked” or skipped all at once.
Access to the required experts is difficult. i.e. there’s a lead time to them and projects are asked to batch their testing, add it to a queue and wait for results. Cross-functional tests are expensive and hard to do in small chunks. |
Frequent Practices | Incremental Nonfunctional Testing (from the start and building upon)Baseline Performance before TuningTest Environments (simulating, emulating or replicating production environments) |
Some frequent journeys
Traditional Programmer
- Q1
- Code along a TDD intro book
- Practice TDD pairing with experts and novices
- Read on advanced techniques
- Practice TDD pairing with experts and novices
- Help others learning TDD
- Q2
- Prototyping
- Wireframes
- Mock-ups
- ATDD + Test Automation
- Read intro book
- Guided by an expert, implement one automated test suite behind-the-GUI
- Experiment with different automation tools
- Work through-the-GUI
- Learn how to write AT specifications
- Use the automation tools beyond AT
- Prototyping
- Q4
- Support an expert by providing application specific info
- Implement Q4 Testing for a simple low-risk case
- Q3
- Support role: Work along test experts in the team providing them with the support they need (i.e. white box testing design)
Traditional Senior Tester
- Read Agile Testing intro book
- Q2
- ATDD, Test Automation
- Read ATDD intro book
- Learn writing AT specs pairing with an agile expert
- Get some hands-on practice using some BDD / ATDD automation tools
- Pair with experts and novices on automation tasks
- After TDD learning, pair with experts on test harness implementation tasks
- ATDD, Test Automation
- Q3
- Agile Exploratory Testing
- Pair with an Agile expert to learn timeboxing skills
- Learn how to use log watching tools
- Learn how to use Test Data Generators
- Design, tool and implement automation that reduces time-consuming tasks (i.e. setup)
- Scenario Development
- Pair with a senior test expert learning to execute already defined Test Scenarios
- Workflow
- Persona-based
- Soap Opera
- Try developing new Scenarios including
- Workflow
- Persona-based
- Soap Opera
- Pair with a senior test expert learning to execute already defined Test Scenarios
- Agile Exploratory Testing
- Q1 – TDD basics
- Acquire basic coding skills and TDD understanding
- Pair with programmers on project tasks (focus on learning xUnit tools and coding)
- Q4 – Support role
- Support an expert by providing application specific info
- Implement Q4 Testing for a simple low-risk case
Traditional Junior Tester
- Read Agile Testing intro book
- Q1 – TDD basics
- Acquire basic coding skills and TDD understanding
- Pair with programmers on project tasks (focus on learning xUnit tools and coding)
- Q2 – ATDD, Test Automation
- Read ATDD intro book
- Pair with experts on automation tasks
- Pair with experts on test harness implementation tasks
- Learn writing AT specs pairing with an ATDD expert
- Pair with experts and novices on Q2 tasks
- Q3
- Pair with an Agile test expert learning how to do Agile exploratory testing
- Pair with a senior test expert learning to execute already defined Test Scenarios
- Workflow
- Persona-based
- Soap Opera
Business Analyst
- Q2 – ATDD, Test Automation
- Read ATDD intro book
- Learn writing AT specs pairing with an agile expert
- Q3
- Pair with an Agile test expert learning how to do Agile exploratory testing
- Pair with a senior test expert learning to execute already defined Test Scenarios
- Workflow
- Persona-based
- Soap Opera
Agile Developer Mid-journey
- Q1
- Read on advanced techniques
- Practice TDD pairing with experts and novices
- Help others learning TDD
- Q2
- Prototyping
- Wireframes
- Mock-ups
- ATDD + Test Automation
- Read intro book
- Guided by an expert, implement one automated test suite behind-the-GUI
- Experiment with different automation tools
- Work through-the-GUI
- Learn how to write AT specifications
- Use the automation tools beyond AT
- Prototyping
- Q4
- Support an expert by providing application specific info
- Implement Q4 Testing for a simple low-risk case
- Q3
- Support role: Work along test experts in the team providing them with the support they need (i.e. white box testing design)
Learning Resources
- General intro to becoming an Agile Tester
- Agile Testing: A Practical Guide for Testers and Agile Teams by: Lisa Crispin; Janet Gregory. http://my.safaribooksonline.com/9780321616944
- Agile Testing and Quality Strategies: Discipline Over Rhetoric by Scott Ambler. http://www.ambysoft.com/essays/agileTesting.html
- Getting started with TDD
- Test-Driven Development by Example by Kent Beck. http://techbus.safaribooksonline.com/0-321-14653-0
- Test-Driven Development: A Practical Guide by David Astels. http://www.amazon.com/exec/obidos/ASIN/0131016490
- Advanced TDD
- xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros. http://www.amazon.com/exec/obidos/ASIN/0131495054/ambysoftinc
- How to move to ATDD
- ATDD by Example: A Practical Guide to Acceptance Test-Driven Development by Markus Gärtner. http://techbus.safaribooksonline.com/book/software-engineering-and-development/agile-development/9780132763219
- Distributed Teams and testing
- Software Testing in Distributed Teams by Liza Crispin. http://www.methodsandtools.com/archive/distributedsoftwaretesting.php
- Intro to Scenario Testing
- An Introduction to Scenario Testing, by Cem Kaner. http://www.kaner.com/pdfs/ScenarioIntroVer4.pdf
- Intro to Soap Opera Testing
- Soap Opera Testing, by Hans Buwalda. http://www.logigear.com/logi_media_dir/Documents/Soap_Opera_Testing.pdf