CODE Coverage Reporting

Andi Lamprecht ·2026-05-11· 3 min read· Accepted

References

Title

Context

Code Coverage Reporting tool is a central place for uploading code coverage reports of unit testing and sourcing the information from to compare the progress across time, attach information to repositories and pipelines as a PR status check and badges to increase deployment frequency, code quality and stability, reduce time to production and incident rates.

Decision

Proceed with CodeCov platform to exploreit for a PoC for central code coverage reports storing and referencing results and history of those on GitHub repositories and pull requests.

Decision drivers

• Simplicity
• Automation
• Scalability
• Security
• Cost

Consequences

Key points for selecting CodeCov to proceed with PoC:

• Ability to track progress over time and compare with some older results
• Ability to configure the gates centrally to prevent PRs from merging, if coverage is lower than X
• Ability to add extended information in PR about codecoverage
• Ability to track metrics across different project/repos
• Upload customization through config file (merging reports, exclude paths, highlight/annotate code in PR)
• SOC2 compliance (Codecov platform will have access to all our code base which requires security compliance)

The consequence of this decision is implementation, which is highlighted by the Cons section of the chosen option. This approach contains a few stages:

1. Sign Up to CodeCov through Github, purchase a license.
2. Install Codecov GitHub app.
3. Setup global token for coverage reports upload.
   Note: requires validation, if it is possible to on-board new repo from scratch using cli/api and global token.
4. Generate codecoverage report (requires steps individually in each repo, will vary based on language).
5. Add a step in CI to upload code coverage report and put a comment on PR.

Alternatives Considered

Official terraform provider isn’t available in any option, so not mentioned in Pros/Cons.

Coveralls

Pros:

• Cost
• CircleCI orb
• Simplicity
• API support for initial repo setup
• GitHub PR checks

Cons:

• Shorter list of languages coverage in comparison with other alternatives
• SOC2 not covered
• No way for admins manitenance

Codecov

Pros:

• Multi-language support
• Security (SOC2 compliant)
• CircleCI Orb
• SSO through GitHub
• Global token for on-boarding new repos
• Focused on code coverage

Cons:

• Manual steps for adding new admins
• Need to validate first time repo on-boarding automation

SonarCloud

Pros:

• Multi-language support
• SSO through GitHub
• Additionally provide SAST reporting on PRs
• Support central configuration for Quality Gates

Cons:

• CircleCI orb only for SAST component, but not for code coverage report uploading
• Manual steps for adding new admins
• Need to validate first time repo on-boarding automation
• Cost is based on lines of code across all repos, can be less predictable
• No SOC2

SonarQube (hosted option from SonarCloud)

Pros:

• SSO with Okta
• Customization
• Ability to add more pre-existing plugins
• Multi-language support
• CapEx

Cons:

• Security
• Hosting platform on our cloud will require additional maintenance overhead
• OpEx

Self-hosting reports

This option will provide a simple approach of collecting code coverage reports in central cloud storage or in CI system test reports (CircleCI), instead of external platform.

Pros:

• Security (no need to expose code base to another platform)
• Simplicity
• Cost

Cons:

• Require allocation/maintenance of storage on internal infrastructure and maintaining access to it from VCS and/or local
• Lack of built-in integrations and features, like PR annotations and comments
• Lack of ability to track progress over time and across different projects

Links

• [Code coverage reporting tool comparison](confluence-title://PE/Code Coverage Reporting tool Evaluation)