Lessons Learnt: Testing in a Changing Context
As a tester, I do my best to follow the principles of Context Driven Testing.
Embarrassingly enough, only up until recently, did I realise that context can change, while you are working in a team - so anything you put in place at the start of your time in a team, might not be the best thing for everyone 6 months later, or 12 months later.
I’d like to share some things I’ve learned about testing in a changing context - but first let me describe my initial context and how it changed over the course of roughly 14-15 months.
Initial Context
- 2 main applications - fairly straightforward, not too many possible flows in either of them
- 3 Backend developers
- 4 Frontend developers
- No test automation set up initially
- 1 Product Analyst
- 1 Product Owner
- Designer, who was not part of our team
- 1 Scrummaster
How the Context Changed
- 7 applications - a few of which had various possible user flows within it. 1 of these applications had many flows in it.
- 3 Backend developers
- 7 Frontend developers
- Cypress Test Automation, written for most of the applications. Running automatically for 2 of the smaller/more straight forward applications. The tests had to be triggered manually for the rest of them (We had Selenium Webdriver tests for a bit, before our Test Automation Specialist decided to focus more on Cypress as a company-wide Test Automation initiative).
- API Test Automation for a few of the applications - had to be triggered manually
- 2 Product Analysts
- 2 Product Owners
- Designer, who was part of our team
- No Scrummaster
Lesson 1. Since Context can change, testing processes and approaches should change
Looking back, I think there was an initially good set-up of how testing was carried out and the strategy that was in place to make sure changes; new features etc were tested thoroughly - this was aided as we introduced Test Automation early on (initially Selenium Webdriver tests).
But then as we had to support more applications, and our team grew - there are some parts I would have addressed sooner rather than later.
Given how the number of developers increased, and the number of applications increased, I wish more time and effort was put into getting the developers to contribute to the test automation instead of just our Test Automation specialist.
It meant, towards the end of me being in the team, a lot of time was spent maintaining the Test Automation suite, but the level of coverage wasn’t quite where we needed it to be, hence a lot of manual regression was still needed after each release.
Lesson 2. There’s a difference between having a Testing Process, that exists, and having one that is followed.
Towards the end of my time in this team, a question came up multiple times of ‘What is the testing process?" Or “Do we even have a testing process?”
This was something we had discussed as a team multiple times, had documented in our wiki, had 1 on 1 discussions about and had been shared/reminded about multiple times.
But I learned the hard way, that even if people say they understand, or they will follow it - it doesn’t necessarily mean that everyone will. This could be due to a lack of perceived understanding (they think they understand, so they don’t ask questions) or a fear of not wanting to say something (don’t want to offend people or don’t want to seem like they don’t know what they are doing).
Looking back, I would’ve been more direct at reminding specific people to follow the process or asking specific people about how they went about testing or what concerns they had at the time. At the time, I was (unfortunately) too worried about offending people or hurting people’s feelings that I always went for a general approach and reminded “everyone” but then now I realise that the people who most needed to be reminded about the testing process (we as a team had agreed one) might not have realised that the reminders to the team didn’t need to equally apply to everyone, and that THEY themselves had to be reminded.
Lesson 3. It can be tricky to learn how to define “ownership of quality of your work” as a developer, when you also have tester(s) in the team
At our project, we were all encouraged to think that “quality is a team effort” or that “everyone owns quality”. Or that “everyone should help with testing”.
But then how are developers supposed to interpret that when you have someone whose role is focussed on manual testing and another focussed on test automation?
Are we supposed to be the safety net? (For example)
I tried to explain it to the developers like this:
Before you give us changes/features etc to test, I expect that you have done some sanity testing to make sure your changes are there as expected, and it doesn’t break anything obvious. I’m here to help find things more creatively. (then I would give some examples of the types of tests I would run)
But then if you are holding the developers responsible for the quality of their changes/features then what role should the tester(s) have in the team? What’s the point in having them in the first place?
I would write testing notes, and discuss my assumptions/ what I would test when the features etc are initially defined - to give the developers the “heads up” of what I would be looking out for, before they start writing any code. The great thing about this, is that if I misunderstood anything, then the developers or the product analyst would correct me.