When writing software, errors are bound to occur in your code. This can be in the form of syntax errors, logic errors, runtime errors, performance errors, data errors, or worse still.
The almost ubiquitous nature of errors while writing software makes software testing an important step in software development. Proper and effective testing not only results in high-quality software that meets users’ needs but also software that is compliant with regulations and has little to no vulnerabilities that can be exploited by attackers.
Software testing can be done in the form of automated testing, unit testing, integration testing, or testing the entire software through system testing.
However, an important that arises during software is how you determine whether your software has been tested comprehensively. Are the tests you’ve run on the software enough?
Have you fully tested all parts of your software, or are there pieces of code that have not been tested at all?
Such questions are bound to arise when doing software testing, and this is what makes code coverage so important.
Code coverage is a software testing metric that tells you how much of your code has been executed and thus tested by the tests you’ve run on the software you’re testing. Code coverage results are expressed as a percentage.
This percentage shows how much of your code has been covered by the tests you’ve run on the code. For instance, if you run a test and get back a code coverage of 60%, it means 40% of your code has not been covered by the tests you’ve written, and thus there could be errors and vulnerabilities in the untested code.
For this reason, code coverage allows you to analyze the effectiveness and completeness of the tests you’ve run on your software. This has the benefit of ensuring software is properly and comprehensively tested before it is released. This reduces software bugs in production software which could affect users’ experience with the software.
As much as a code coverage of 100% does not necessarily mean that the software you’re writing is totally error-free, you want to aim for high code coverage to ensure efficient testing of your software.
In critical industries such as aerospace and medicine, where software errors could result in death, regulations demand 100% software coverage during testing.
Types of Code Coverage Metrics
There are several types of code coverage metrics that can be measured while testing software. They include:
- Statement Coverage – measures the percentage of executable statements in the source code which have been executed during testing.
- Function Coverage – measures the percentage of defined functions that have been called during testing.
- Branch Coverage – measures the percentage of branches or possible paths that have been executed from all the decision points in the source code. It is used to ensure that all branches that arise from decision control structures such as if, switch statements, and if else statements have been fully tested.
- Condition coverage – measures the percentage of boolean expressions that have been tested for both true and false values.
- Loop Coverage – measures the percentage of loops in the source code that have been executed during testing.
- Path Coverage – measures the percentage of all possible execution paths in the source code that have been tested.
The above metrics are usually included in a code coverage report.
Code Coverage Best Practices
There are certain best practices that are recommended while conducting code coverage to ensure the effectiveness and quality of the code coverage. They include:
Have clear coverage goals
In any software testing undertaken, set target coverage percentages for each test coverage metric that is appropriate. This has the benefit of not only providing clear testing targets but also helping to reduce defects in software by directing team efforts in increasing code coverage. It also helps in ensuring that software testing is given the attention it deserves during software development.
Focus on Test Quality
It is important to note that code coverage simply shows the percentage of code that has been tested and does not show whether they have been tested correctly or whether the software is bug-free. Therefore rather than simply focusing on getting code coverage closer to 100 percent, the emphasis should be on writing quality and effective tests that correctly test the software and add value.
Increase code coverage in frequently changing code
Whereas achieving high code coverage numbers in large projects might be difficult, efforts can be put in to ensure that the code coverage gets better over time.
A great way to do this is by requiring high code coverage of over 90 percent in every new commit being made to the project code base.
Enforcing commit-level code coverage is not only realistic and feasible but also ensures that any new changes made to the software have excellent code coverage.
Measure and analyze code coverage data
Utilize the results gotten from code coverage to identify areas that still need testing and to also guide future testing efforts with a priority on areas with low code coverage.
Analyze code coverage data to identify critical areas of an application that is yet to be tested and direct your efforts to fully test the untested critical areas. Using code coverage data to improve and prioritize software results in better-tested software with fewer defects.
Code Coverage vs. Test Coverage
Whereas both are used to ascertain the effectiveness of tests, code coverage and test coverage are fundamentally different in their use and what they measure.
Test Coverage is a metric used to determine the extent to which the written tests cover the requirements of the software. It involves testing each software requirement, and it helps ascertain how well software has been tested in regards to its meeting its requirements.
Test coverage results show the percentage of the software requirements that have been tested. Test coverage is typically done by quality assurance professionals.
Code coverage, on the other hand, is a software testing metric used to determine the percentage of the source code that has been executed by the written tests.
The results of a code coverage show the extent to which statements, functions, paths, loops, branches, and conditions in the source code have been executed by the written unit tests. Code coverage is used to evaluate how well the tests written cover the source code and is typically done by software developers.
Code coverage is an important metric to measure while doing software testing. Here are some tools to help you with code coverage:
Clover
Clover is an open-source code coverage tool initially developed by Atlassian, an Australian software company that develops products for software development teams. The tool is written purely in Java and can run on any operating system that satisfies the requirements for the Java Runtime Environment.
Clover can be used to perform code coverage on code written in Java, Groovy, or AspectJ programming languages. It has support for test frameworks such as JUnit, TestNG, and Spock, and it can also be integrated with IDEs such as IntelliJ IDEA and Eclipse.
Clover can be used to measure code coverage metrics such as method, statement, branch, global and per-test coverage.
From a code coverage exercise, it can generate highly configurable HTML reports that show the code coverage results in addition to the top risk areas in the software and can be used in test optimizations.
The reports can also be generated in PDF, XML, JSON, or plain text. Key advantages of clover are that it can be integrated with many different tools and it is actively being developed and improved upon.
JaCoCo
JaCoCo is a free code coverage library for the Java programming language developed by the EclEmma team. The library is implemented in EclEmma, which is a free Java code coverage tool for the Eclipse IDE.
JaCoCo provides a rich coverage analysis whose results are immediately summarized and highlighted in the Java Source code editor and allows users to drill down the coverage results to the method level.
The results are presented using a customizable color code that highlights the lines of code that have been fully, partly, or not yet covered by tests run on the source code. It allows for the merging and consideration of different test runs in order to arrive at a total code coverage of the source code.
JaCoCo is a lightweight tool and does not require modifying your projects or performing any other setups to use for code coverage analysis.
Cobertura
Cobertura is a free and open-source Java code coverage tool that is based on Jcoverage and can be used on its own, through Ant script, or through the Maven plugin. Using it through a Maven plugin is the most common way to use Cobertura for code coverage.
Cobertura measures the percentage of lines or branches that have been executed by tests run on a Java source code. It provides metrics such as line coverage which shows the percentage of statements executed during tests, and also branch coverage which shows the percentage of branches covered during tests.
It also shows a complexity factor that increases as the number of branches in your Java code increases.
The code coverage results are presented in HTML or XML, showing which parts of the source code have not been tested. Aside from showing test coverage results, Cobertura can also be used in locating untested code and bugs and also identifying unreachable code.
Istanbul
Istanbul is a code coverage tool for JavaScript code with support for ES6+. This tool can be installed in any Javascript project as a development dependency using the node package manager.
Istanbul provides code coverage metrics such as statement, branch, function, and line coverage. It also shows the lines in the source code that have not been covered by the tests. It does this by adding line counters to your JavaScript code so that it can track the extent to which your unit tests execute your source code.
Code coverage results by Istanbul can be output in the terminal or in the form of HTML. Additionally, Istanbul offers support for applications that spawn subprocesses, source mapped coverage of Babel and TypeScript projects.
Pytest-cov
Pytest-cov is a free Python plugin used to generate code coverage reports for Python code. It is installed using Python’s package installer Pip and is operated from the command line.
Its code coverage report shows the statements in your Python project, those not covered by tests, and it provides a test coverage percentage showing the percentage of your Python code covered by tests.
Pytest-cov offers subprocess support, xdist support, and consistent pytest behavior. Pytest-cov’s default behavior while conducting tests is deleting existing coverage data files to ensure new and clean data for each new test run. However, it also allows users to combine the code coverage test results from previous test runs.
Coverage.py
Coverage.py is a code coverage tool for Python programs, and it is installed in projects using pip.
By default, it measures line or statement coverage and provides results showing the number of statements in the program, those missed by tests, and the percentage coverage from the test, and it also shows the lines in your Python source code that have been missed by the tests. However, it can still be configured to measure branch coverage in Python programs.
Coverage.py can also be used to tell which tests ran which lines in the source code. Its code coverage report can be presented in the terminal and also in HTML, XML, JSON, and LCOV formats.
SimpleCov
SimpleCov is a robust code coverage tool for the Ruby programming language. It utilizes Ruby’s built-in coverage library to gather relevant data to be used in determining the code coverage after running tests.
The best thing about SimpleCov is its presentation of code coverage results. It also merges results from different types of tests done so that the report generated shows the results from all the tests done, allowing the easy identification of untested parts of the code.
It also formats the source code with color codes which can easily be used to identify tested and untested parts of the code. By default, SimpleCov measures and reports the line coverage of tests. However, it can be configured to measure and report on the branch coverage of the performed tests.
Deep Cover
Deep Cover is an accurate code coverage tool for Ruby code. It offers more accurate line coverage reports by ensuring that a line is considered covered only when it is executed entirely and not partially.
Additionally, it offers support for node and branch coverage which can optionally be used to find out if there are some branches not taken by the tests.
Deep Cover is not only easy to use without the need for configurations, but it can be integrated into projects using other code coverage tools such as Ruby’s built-in code coverage library or SimpleCov. In such cases, Deep Cover makes the tools stricter by only marking lines as executed only if everything on the line of code is fully executed.
Conclusion
As much as higher code coverage will not necessarily result in error-free software, it is a crucial metric that needs to be factored in when testing software. Code coverage is important in evaluating how much the tests written actually test a software’s source code.
Additionally, working on improving the code coverage during testing results in better-tested software that is less prone to errors in production. To perform code coverage while testing software, consider using the tools suggested in the article.