Skip to content

Add Code Coverage Data Collected page #30830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5133,6 +5133,11 @@ menu:
parent: code_coverage
identifier: code_coverage_setup
weight: 1
- name: Data Collected
url: code_coverage/data_collected/
parent: code_coverage
identifier: code_coverage_data_collected
weight: 2
- name: Quality Gates
url: quality_gates/
pre: ci
Expand Down
3 changes: 3 additions & 0 deletions content/en/code_coverage/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ further_reading:
- link: "/code_coverage/setup"
tag: "Documentation"
text: "Set up Code Coverage"
- link: "/code_coverage/data_collected"
tag: "Documentation"
text: "Learn what data is collected for Code Coverage"
algolia:
tags: ["code coverage"]
---
Expand Down
56 changes: 56 additions & 0 deletions content/en/code_coverage/data_collected.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
---
title: Code Coverage Data Collected
further_reading:
- link: "/code_coverage"
tag: "Documentation"
text: "Code Coverage"
- link: "/code_coverage/setup"
tag: "Documentation"
text: "Set up Code Coverage"
---

{{< callout url="http://datadoghq.com/product-preview/code-coverage/" >}}
Code Coverage is in Preview. This product replaces Test Optimization's <a href="https://docs.datadoghq.com/tests/code_coverage">code coverage</a> feature, which is being deprecated. Complete the form to request access for the new Code Coverage product.
{{< /callout >}}

## Source Code Provider Integration

The exact data received by Datadog depends on your source code provider type:

{{< tabs >}}
{{% tab "GitHub" %}}

Code Coverage relies on the following GitHub webhooks:
* Pull request
* Pull request review
* Pull request review comment
* Push

None of the webhooks include your source code content; they only include metadata about the pull request, such as title, description, author, labels, and commit SHAs.

See GitHub's [webhook events and payloads documentation][1] for a detailed description of the data sent by these webhooks.

[1]: https://docs.github.com/en/webhooks/webhook-events-and-payloads

{{% /tab %}}
{{< /tabs >}}

By default, when synchronizing your repositories, Datadog doesn’t store the actual content of files in your repository, only the Git commit and tree objects.

See [Datadog Source Code Integration][1] for more information about how Datadog integrates with your source code provider.

## Code Coverage Report Upload

The `datadog-ci coverage upload` command sends the following data to Datadog:
- **Coverage reports**: The report files, which contain the coverage data for your codebase. The data depends on the coverage tool and report format you are using, and normally includes file paths, line numbers, and coverage percentages.
- **Git metadata**: Git repository URL, branch name, commit SHA, timestamp, author information, and list of file paths that were changed in the commit. You can disable Git metadata upload by adding `--skip-git-metadata-upload=1` to the command.
- **Git diff summary**: List of file paths that were changed in the commit, along with the numbers of added and removed lines. You can disable Diff data upload by adding `--upload-git-diff=0` to the command.
- **CI metadata**: Information about the CI environment, such as the CI provider, job ID, and pipeline ID.

No source code is uploaded to Datadog.

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /integrations/guide/source-code-integration
48 changes: 30 additions & 18 deletions content/en/code_coverage/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ further_reading:
- link: "/code_coverage"
tag: "Documentation"
text: "Code Coverage"
- link: "/code_coverage/data_collected"
tag: "Documentation"
text: "Learn what data is collected for Code Coverage"
---

{{< callout url="http://datadoghq.com/product-preview/code-coverage/" >}}
Expand Down Expand Up @@ -41,28 +44,36 @@ The following webhooks are required:
| Pull request review comment | Receive PR data updates. |
| Push | Receive Git commit metadata. |

If everything is configured correctly, a green check mark is displayed in Datadog's [GitHub Integration][2] page:
{{< img src="/code_coverage/github_app_success.png" alt="GitHub App integration success check" style="width:100%" >}}

<div class="alert alert-info">If you have a Datadog-managed Marketplace App or a custom app with default settings, the required permissions and webhooks are included.</div>

[1]: /integrations/github/#github-apps-1
[2]: https://app.datadoghq.com/integrations/github/configuration
{{% /tab %}}
{{< /tabs >}}

See [Data Collected][1] for details on what data is collected from your source code provider.

## Data access permissions

If you are using [custom roles][1] rather than [Datadog-managed roles][2], be sure to enable the `Code Coverage Read` permission for the roles that need to view code coverage data.
If you are using [custom roles][2] rather than [Datadog-managed roles][3], be sure to enable the `Code Coverage Read` permission for the roles that need to view code coverage data.

Navigate to [Roles settings][3], click `Edit` on the role you need, add the `Code Coverage Read` permission to the role, and save the changes.
Navigate to [Roles settings][4], click `Edit` on the role you need, add the `Code Coverage Read` permission to the role, and save the changes.

## PR Gates

If you wish to gate on PR coverage, configure PR Gates rules in Datadog.

Navigate to [PR Gates rule creation][4] and configure a rule to gate on total or patch coverage.
Navigate to [PR Gates rule creation][5] and configure a rule to gate on total or patch coverage.

## Upload code coverage reports

Update your CI pipeline to upload code coverage report files to Datadog. This involves installing and running the `datadog-ci` CLI in your CI environment.

See [Data Collected][6] for details on what data is collected during code coverage report upload.

### Supported coverage report formats

Datadog supports the following coverage data formats—expand for examples:
Expand Down Expand Up @@ -238,15 +249,15 @@ end_of_record

### Install the datadog-ci CLI

Install the [`datadog-ci`][5] CLI globally using `npm`:
Install the [`datadog-ci`][7] CLI globally using `npm`:

{{< code-block lang="shell" >}}
npm install -g @datadog/datadog-ci
{{< /code-block >}}

#### Standalone binary

If installing Node.js in the CI is an issue, standalone binaries are provided with [Datadog CI releases][6]. Only _linux-x64_, _linux-arm64_, _darwin-x64_, _darwin-arm64_ (MacOS) and _win-x64_ (Windows) are supported. To install, run the following from your terminal:
If installing Node.js in the CI is an issue, standalone binaries are provided with [Datadog CI releases][8]. Only _linux-x64_, _linux-arm64_, _darwin-x64_, _darwin-arm64_ (MacOS) and _win-x64_ (Windows) are supported. To install, run the following from your terminal:

{{< tabs >}}
{{% tab "Linux" %}}
Expand Down Expand Up @@ -285,7 +296,7 @@ Start-Process -FilePath "./datadog-ci.exe" -ArgumentList version

### Uploading coverage reports

To upload your code coverage reports to Datadog, run the following command. Provide a valid [Datadog API key][7] (`DD_API_KEY`), and one or more file paths to either the coverage report files directly or directories containing them:
To upload your code coverage reports to Datadog, run the following command. Provide a valid [Datadog API key][9] (`DD_API_KEY`), and one or more file paths to either the coverage report files directly or directories containing them:

{{< tabs >}}
{{% tab "GitHub Actions" %}}
Expand All @@ -299,28 +310,29 @@ steps:
DD_SITE: {{< region-param key="dd_site" >}}
</code>
</pre>
[1]: https://docs.github.com/en/actions/learn-github-actions/expressions#always
{{% /tab %}}
{{< /tabs >}}

The command recursively searches the specified directories for supported coverage report files, so specifying the current directory (`.`) is usually sufficient.
See the [`datadog-ci` documentation][8] for more details on the `datadog-ci coverage upload` command.
See the [`datadog-ci` documentation][10] for more details on the `datadog-ci coverage upload` command.

Shortly after the code coverage report upload is finished, Datadog adds a PR comment with code coverage percentage values.
You can also view your coverage data aggregated by pull request in the [Code Coverage page][9] in Datadog, with the ability to examine individual files and lines of code.
You can also view your coverage data aggregated by pull request in the [Code Coverage page][11] in Datadog, with the ability to examine individual files and lines of code.

{{< img src="/code_coverage/pr_details.png" text="Code Coverage PR details page in Datadog" style="width:100%" >}}

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /account_management/rbac/permissions/#custom-roles
[2]: /account_management/rbac/permissions/#managed-roles
[3]: https://app.datadoghq.com/organization-settings/roles
[4]: https://app.datadoghq.com/ci/pr-gates/rule/create
[5]: https://www.npmjs.com/package/@datadog/datadog-ci
[6]: https://github.com/DataDog/datadog-ci/releases
[7]: https://app.datadoghq.com/organization-settings/api-keys
[8]: https://github.com/DataDog/datadog-ci/blob/master/src/commands/coverage/README.md
[9]: https://app.datadoghq.com/ci/code-coverage
[1]: /code_coverage/data_collected/#source-code-provider-integration
[2]: /account_management/rbac/permissions/#custom-roles
[3]: /account_management/rbac/permissions/#managed-roles
[4]: https://app.datadoghq.com/organization-settings/roles
[5]: https://app.datadoghq.com/ci/pr-gates/rule/create
[6]: /code_coverage/data_collected/#code-coverage-report-upload
[7]: https://www.npmjs.com/package/@datadog/datadog-ci
[8]: https://github.com/DataDog/datadog-ci/releases
[9]: https://app.datadoghq.com/organization-settings/api-keys
[10]: https://github.com/DataDog/datadog-ci/blob/master/src/commands/coverage/README.md
[11]: https://app.datadoghq.com/ci/code-coverage
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading