Scrum minus testing plan equals POC on steroids.
Nowadays, most successful projects start with a Proof of Concept (POC), a relatively small-sized assessment of an idea, where the chosen technology and feature pack will be verified first, evaluated for potential impact on business users, and then, once proven worthy of investment, a full-sized project team is assigned to design and deliver the full-featured product and deploy it to production.
This is the perfect case for a Scrum team. The team can rapidly develop the prototype, adding substantial new features each sprint, while business users can watch in real-time fast progress and how the idea is built from scratch within just about 10 sprints. It leaves a strong impression (which is the main target of the POC anyway), but it has also one significant property – minimal or no testing activities, and even thinking about the testing process would be a straight joke.
It is no fun activity inside a Scrum team, and people will mostly like the idea of developing without processes around to slow them down. This is basically what any testing activity will do implicitly. And who does want unnecessary slowness during the time we need to impress the end user the most?
The state that happens if the project team continues in the same manner after the POC period is done is something I use to call POC on steroids – a production system growing in size and features, but still behaving like a POC – a wannabee complete product with hidden defects and unexplored corner cases, only waiting for a serious crash.
But before it converts there, the team will have a harder time from sprint to sprint keeping up with stable releases, as fixing the rolling issues will become only more complex.
Here are some techniques that proved to be working when I was dealing with similar problems and which I believe can be named as best practices for implementing solid testing processes, still without cluttering the development progress too much – because that’s what we all want to be for a Scrum team.
Distribute the pain
Where else shall we start when dealing with unnecessary bothering than with splitting the pain :).
And what I mean by this is creating a plan that will require small additional activity from developers here and there but that will contribute to the common target greatly over time, incrementally but consistently.
#1. Develop Unit test code for each created piece of new code
If you manage to convince your scrum teams to add into its Definition Of Done clause development of unit tests for each new code created within the sprint, then from a long-term perspective, this will be a great achievement.
The reasons are fairly obvious:
It will force developers to think about various non-standard paths of the code. –
- Such unit tests can be plugged into automated DevOps pipelines and executed with every single deployment to the dev or test environment. Also, metrics from the pipeline can be easily exported and then used for demonstration to business users the percentual coverage of test cases directly bound to the source code.
The most important is to start very soon. The later the unit tests will start to be developed, the more distracting it will become for developers to add them within a sprint.
- It will take significant effort to backward develop unit tests for already existing code. Some parts of the code might be created by another developer and the exact knowledge of how it should work in every single code part is not exactly clear to the current developer. In some cases, it can come so far that adding a unit test to the modified code will take more time than developing the feature change for the sprint (which is a state nobody calculated during sprint planning for sure).
#2. Make a routine of executing all parts of Unit Tests on the Development environment
Even before creating a pull request for merging the new code into the Master branch, it shall be a standard activity to test both the feature code and the unit test code on the dev environment. By doing this it will be ensured that:
- Unit test code is actually working for each part (in the end it is just another code that must be verified). This step is very often totally skipped. It is somehow assumed that if the unit test is anyway plugged into the DevOps pipeline, it will be eventually executed and tested somewhere by default. However, this is nothing but pushing the problems into upper levels of sprint activities, where the team has usually less time and more stress to finish every committed story.
- The new feature code is tested by the developer for basic functionality. Obviously, it can’t be expected from this test that the business functionality will be fully verified, but at least this test will confirm the code behaves in the way the developer intended it (ignoring, for now, the fact that the business logic could be incorrectly understood while developing).
#3. Expect unit test execution after code merge into the master branch
It is one thing to have functional code on the local branch (where nobody but the branch owner is developing a new feature), but it is a completely different thing to have the same code working after the pull request and merge into the master branch.
The master branch contains changes from other Scrum team members, and even if the conflicts are resolved and everything looks fine, the code after merge and conflicts resolution is basically still an untested piece of code, and it is risky to send it further without verifying first it is still working fine.
So what proved to be effective here is just simply asking to execute the same unit testing – already done previously on the dev environment – also on the environment, which is built from the master branch code version.
For the developers, it might be one additional step they need to include into their life, but it is not that much of an effort since, in this case, nothing new must be invented – just re-execute what was already done once.
Now, this step could be skipped in one particular case:
- The automated unit tests in DevOps pipelines are so comprehensive that they already cover the whole manual testing executed on top of that.
While achieving this state is definitely feasible, I never experienced such a state in real life. It would be probably even too time-consuming for developers to create such deeply detailed automated Unit tests. It could get easily unacceptable for the product owner to let the team dedicate so much time to this activity, as it would have an explicit impact on the number of delivered stories within a sprint.
However, the preference for sprint content shall never be an excuse for not doing the unit tests, or even minimizing them. By doing this, the team will convert again to such a state the code is lacking too much unit test coverage, and then for a catch-up, it might be already too late (again, the higher effort for unit test completion than the actual code change for a sprint).
For all those reasons, I would recommend re-execution of the unit tests on the master code version without hesitations. It is such a low effort compared to what value it will bring.
It will make the final check for the readiness of the master branch for the release testing phase. Also, it will help to find most of the technical bugs (e.g. bugs that prevent the source code from being successfully executed till the successful end), leaving for the next phase to concentrate solely on verification of the business functionality.
Prepare for Functional Testing
All the previous testing activities shall lead to one specific conclusion – the master branch code is free from technical bugs and executable without problems for end-to-end functional flows.
This is a phase that can be easily covered just by a single person, and this person even does not need to have a technical background.
Actually, it is much better if this is executed by someone disconnected from the developers but with functional awareness of what business users expect the code to behave like. There are two main activities to be completed:
#1. New sprint stories targeted functional test
Each sprint shall ideally bring some new functionality (sprint goal increment) that was previously non-existent, and thus it shall be verified. It is important to ensure the new piece of software works to such an extent the business users are happy to have it now, as they were obviously looking forward to it the whole last sprint at minimum :).
It’s such a sad experience when the (long) promised functionality is finally announced to be released, only to find out the other day it actually does not work well.
Therefore, properly testing new sprint functionality is crucial. In order to make this a success, it is good practice to gather important testing cases in advance and from relevant stakeholders (either from the product owner or even from the end users) and make a list of all such test cases needed for the content inside the sprint.
This might look overwhelming at first sight, but from my experience, it is again totally manageable by a single person. Since the sprints are mostly quite short (like two weeks period short), it’s almost never too much new content anyway, as the sprint also contains additional activities like technical debt stories, documentation, design/spike stories, etc.
Test cases are then executed with the aim of verifying the desired functionality foremostly. If any problem occurs with the results, only the relevant developer is approached (the one who owns the changes related to this particular defect) to resolve the problem hopefully quickly.
In this way, the developers will spend minimum time connected with functional testing and can still concentrate on the activities they like the most.
#2. Regression test cases execution
The other part of functional testing shall be to ensure everything that worked until now will also work after the next release. This is what regression tests are for.
Test cases for regression tests are best if they are regularly maintained and revisited before each release. Based on the concrete project specifics, it is best to keep them simple yet cover the majority of the very core functionalities and important end-to-end flows running through the whole system.
Usually, each system has such processes that are touching many different areas, those are the best candidates for regression test cases.
In some cases, regression tests are even implicitly covering also new features in the sprint, for example, if the new story in the sprint is changing some particular part of the existing flow.
So in most cases, it is not at all very complex to complete regression tests alongside new sprint stories functionality tests, especially if this is done regularly before each production release, and the periodicity of production releases is not too small.
Insist on executing Quality Assurance tests before every production release
The Quality Assurance (QA) test is the final step before releasing into production, and often this test is skipped as unimportant. Especially if the scrum team is managed aggressively for new content.
Even business users will say they are interested in new features and not so much in preserving the functionality or keeping the number of defects low. But then again – over time, this is the main reason why the dev team will have to slow down eventually, and then business users will not get what they ask for anyway.
QA test shall be executed solely by the people outside of the Scrum team, ideally by business users themselves in a dedicated environment, which is as close to future production as possible. Alternatively, the product owner can substitute here the end users.
In any case, this should be a clean, functional test from the perspective of the end user, without any connection to the dev Scrum team. It will present one final view of the product and be used in a way possibly nobody from the scrum team anticipated (not at all an ideal case, but it definitely can happen). There is still time for last-minute corrections.
Alternatively, it might become clear the expectations were not well understood by the scrum team, and in such case, at least we can agree on a follow-up plan, still way before the actual production release. This is, again not an ideal case, but still much better like admitting the failure just after reporting a successful production release.
Where to go next from here?
Applying those practices to daily scrum work can seriously bring you to more stabilized and predictable sprint release habits, without delaying the production releases or spending whole sprint just preparing for the next release, or even without forcing everyone in the scrum team to do something they don’t really like or don’t know how to do effectively anyway for the majority of the sprint, that is.
But surely you don’t need to stop here.
Performance test inclusion is one topic to name here. Those are often ignored or deemed unnecessary, scraping in the first round of “scrum activities optimization”. I was however always in doubt about how one can expect the production system to evolve over time if it is not regularly checked for performance.
Going one level up also means the introduction of automated tests.
I have already covered one group of automated tests (unit tests in the pipelines). However, you can develop full end-to-end regression tests completely automated and executed by themselves after every deployment to the test environment. This would free up the development scrum team even more, but it requires a dedicated team to develop and maintain such automated tests. It would become constant work, as whenever production code changes, some existing automated tests can become invalid, and so they must be updated to continue working in the pipelines. It is an effort only a few are willing to pay for, the benefits for the dev scrum team would be, however, great.
Those are topics reaching far above this article’s scope and figuring out the right schedule and timing for each test type mentioned here – so that you can make it within a sprint window. I’ll be happy to dive in next time!