Skip to content

Rfc/privileged rshell helper - #55731

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 38 commits into
mainfrom
rfc/privileged-rshell-helper
Sep 4, 2026
Merged

Rfc/privileged rshell helper#55731
gh-worker-dd-mergequeue-cf854d[bot] merged 38 commits into
mainfrom
rfc/privileged-rshell-helper

Conversation

@astuyve

@astuyve astuyve commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds an opt-in, Linux-only privileged execution path for Private Action Runner remediation commands.

  • Forwards the original signed task envelope, verification key, and TUF Director proof to a socket-activated rshell helper instead of elevating PAR itself.
  • Adds private_action_runner.restricted_shell.privileged.enabled and .socket; privileged execution is limited to remediation tasks marked EscalationAllowed, while whole-script root execution remains rejected.
  • Packages the pure-Go rshell helper as a root-owned Agent binary and installs stable/experiment systemd service and socket units only on amd64/arm64 hosts with Landlock ABI 3+.
  • Adds the rshell build task and standalone bundle script.
  • Adds unit coverage for TUF proof propagation and verification, privileged dispatch, configuration wiring, installer platform gating and unit lifecycle, and generated embedded units.
  • Keeps generated systemd fixture paths short enough for Windows checkouts.

Motivation

Some remediation commands require narrowly scoped root access, but the Private Action Runner should remain unprivileged and must not become the authorization boundary. The helper independently authenticates the backend-signed task, retains only the controlled per-command elevation path, and relies on rshell's Landlock/seccomp worker sandbox. Unsupported hosts and disabled configurations continue using the existing unprivileged path.

Describe how you validated your changes

  • Added and updated Go unit tests around Remote Config TUF proofs, key management, privileged request construction, configuration transforms, installer behavior, and embedded template generation and lookup.
  • Ran bazelisk test //pkg/fleet/installer/packages/embedded:tmpl_test //pkg/fleet/installer/packages/embedded:embedded_test.
  • Ran dda inv -- -e linter.filenames.
  • Ran tools/build-privileged-rshell-agent.sh end to end and verified the three binaries plus generated systemd service/socket contents and modes.
  • Commit hooks passed, including shellcheck, Go formatting, generated systemd-unit validation, and copyright checks.

Additional Notes

  • Linux only; supported architectures are amd64 and arm64.
  • Requires Landlock ABI 3 or newer.
  • The helper is opt-in and the local policy file is optional; when present, it can only narrow backend-signed permissions.
  • Companion implementation: Rfc/privileged rshell helper rshell#605.
  • The initial CI rerun hit a DDCI/GitLab pipeline-creation HTTP 500 before any jobs executed; it was not a test failure.

AI assistance disclosure: This PR was developed with assistance from OpenAI Codex. The author reviewed and validated the resulting code and documentation.

@astuyve
astuyve requested review from a team as code owners September 1, 2026 20:56
@astuyve
astuyve requested review from clarkb7, dd-gplassard, pducolin and s-alad and removed request for a team September 1, 2026 20:56

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The package lets dd-agent change a program that systemd runs as root. It also puts the local policy in a dd-agent-writable directory, omits the socket units from the package lifecycle, and breaks RC client mocks.

Open Bits AI session

🤖 Datadog Autotest · Commit 6b92ef3 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread packages/agent/product/BUILD.bazel Outdated
srcs = [
"@rshell_binary//:rshell",
],
attributes = pkg_attributes(mode = "755"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Keep the root helper owned by root

A process with dd-agent rights can change code that systemd runs as root.

Assertion details
  • Input: Install the Linux package and activate the privileged rshell service.
  • Expected: The installer must keep the helper owned by root and not writable by dd-agent.
  • Actual: The installer changes the helper owner to dd-agent. The service runs this helper as root.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session

@astuyve astuyve Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed: the installer now reapplies root:root ownership to embedded/bin/rshell after the recursive package ownership pass, so dd-agent cannot replace the helper that systemd executes as root.

Type=simple
Group=dd-agent
UMask=0077
ExecStart={{.InstallDir}}/embedded/bin/rshell privileged-helper --policy={{.EtcDir}}/rshell-privileged-helper-policy.json --idle-timeout=30s

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Put the policy in a root-owned directory

A process with dd-agent rights can remove local administrator restrictions.

Assertion details
  • Input: An administrator creates the optional root-owned policy in /etc/datadog-agent.
  • Expected: Store the policy in a root-owned directory that dd-agent cannot change.
  • Actual: The service reads the optional policy from a directory that the installer assigns to dd-agent. The dd-agent user can remove the policy before socket activation.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session

@astuyve astuyve Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed: the helper policy now lives at /etc/datadog-agent/rshell.d/policy.json. The installer creates rshell.d as root:root with mode 0755 and reapplies root ownership recursively after the /etc/datadog-agent dd-agent ownership pass. A regression test checks both the directory state and permission ordering.

"datadog-agent-action-exp.service": mustReadSystemdUnit("datadog-agent-action.service", expData, ambiantCapabilitiesSupported),
"datadog-agent-rshell-privileged.service": mustReadSystemdUnit("datadog-agent-rshell-privileged.service", stableData, ambiantCapabilitiesSupported),
"datadog-agent-rshell-privileged-exp.service": mustReadSystemdUnit("datadog-agent-rshell-privileged.service", expData, ambiantCapabilitiesSupported),
"datadog-agent-rshell-privileged.socket": mustReadSystemdUnit("datadog-agent-rshell-privileged.socket", stableData, ambiantCapabilitiesSupported),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Register the helper units

Privileged remote shell cannot connect to its socket after package installation.

Assertion details
  • Input: Install a DEB, RPM, or OCI package and enable privileged remote shell.
  • Expected: Add both helper service and socket units to the stable and experiment lifecycle lists.
  • Actual: The generated unit map contains the new socket units. The package lifecycle lists omit them, so installation does not write or start them.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed: both the privileged helper service and socket are registered in the stable and experiment Systemd and Procmgr lifecycle lists. The focused installer and embedded-unit tests pass.

// Component can also automatically subscribe to updates by returning a `ListenerProvider` struct
Subscribe(product data.Product, fn func(update map[string]state.RawConfig, applyStateCallback func(string, state.ApplyStatus)))
// GetConfigTUFProof returns the Director proof for one Remote Config target.
GetConfigTUFProof(targetPath string) (state.ConfigTUFProof, bool)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Update all RC client implementations

Affected test packages fail to compile and block CI.

Assertion details
  • Input: Compile a test package that assigns an existing mock to rcclient.Component.
  • Expected: Update all RC client implementations, or use a smaller interface for proof access.
  • Actual: Existing mock RC clients do not implement GetConfigTUFProof. Packages that use these mocks do not compile.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session

@astuyve astuyve Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed: the production RC component and adapter implement GetConfigTUFProof, and all five test doubles implementing rcclient.Component now provide the method as well. Focused tests for rcclient, queryactions, datasecurity, datastreams, SNMP check config, eBPF, and private action runner all pass.

@dd-octo-sts

dd-octo-sts Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Go Package Import Differences

Baseline: bfea996
Comparison: 1199dcf

binaryosarchchange
privateactionrunnerlinuxamd64
+3, -0
+github.com/DataDog/rshell/builtins/awk
+github.com/DataDog/rshell/builtins/sha256sum
+github.com/DataDog/rshell/privilegedhelper
privateactionrunnerlinuxarm64
+3, -0
+github.com/DataDog/rshell/builtins/awk
+github.com/DataDog/rshell/builtins/sha256sum
+github.com/DataDog/rshell/privilegedhelper
privateactionrunnerwindowsamd64
+3, -0
+github.com/DataDog/rshell/builtins/awk
+github.com/DataDog/rshell/builtins/sha256sum
+github.com/DataDog/rshell/privilegedhelper
privateactionrunnerdarwinamd64
+3, -0
+github.com/DataDog/rshell/builtins/awk
+github.com/DataDog/rshell/builtins/sha256sum
+github.com/DataDog/rshell/privilegedhelper
privateactionrunnerdarwinarm64
+3, -0
+github.com/DataDog/rshell/builtins/awk
+github.com/DataDog/rshell/builtins/sha256sum
+github.com/DataDog/rshell/privilegedhelper

…-helper

# Conflicts:
#	LICENSE-3rdparty.csv
#	go.mod
#	go.sum
#	pkg/config/schema/yaml/private_action_runner.yaml
#	pkg/config/setup/privateactionrunner.go
#	pkg/config/setup/privateactionrunner_settings.go
#	pkg/fleet/installer/packages/embedded/BUILD.bazel
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm-nc/datadog-agent-rshell-privileged-exp.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm-nc/datadog-agent-rshell-privileged-exp.socket
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm-nc/datadog-agent-rshell-privileged.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm-nc/datadog-agent-rshell-privileged.socket
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm/datadog-agent-rshell-privileged-exp.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm/datadog-agent-rshell-privileged-exp.socket
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm/datadog-agent-rshell-privileged.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/debrpm/datadog-agent-rshell-privileged.socket
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci-nc/datadog-agent-rshell-privileged-exp.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci-nc/datadog-agent-rshell-privileged-exp.socket
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci-nc/datadog-agent-rshell-privileged.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci-nc/datadog-agent-rshell-privileged.socket
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci/datadog-agent-rshell-privileged-exp.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci/datadog-agent-rshell-privileged-exp.socket
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci/datadog-agent-rshell-privileged.service
#	pkg/fleet/installer/packages/embedded/tmpl/gen/sd/oci/datadog-agent-rshell-privileged.socket
#	pkg/fleet/installer/packages/embedded/tmpl/main.go
#	pkg/privateactionrunner/adapters/config/config_adapter.go
#	pkg/privateactionrunner/adapters/config/transform.go
#	pkg/privateactionrunner/bundles/remoteaction/rshell/BUILD.bazel
#	pkg/privateactionrunner/bundles/remoteaction/rshell/entrypoint.go
#	pkg/privateactionrunner/bundles/remoteaction/rshell/run_command.go
#	pkg/privateactionrunner/task-verifier/keys_manager.go
@dd-octo-sts

dd-octo-sts Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor bfea996
📊 Static Quality Gates Dashboard
🔗 SQG Job
⚠️ Exception granted by @chouquette: this PR exceeds the per-PR size thresholds but will not be blocked.

Error

Quality gate Change Size (prev → curr → max)
⚠️ agent_deb_amd64 (per-PR threshold) +11.81 MiB (1.54% increase, -81.64% of buffer) 768.471 → 780.283 → 782.940
⚠️ agent_rpm_amd64 (per-PR threshold) +11.81 MiB (1.54% increase, -81.71% of buffer) 768.455 → 780.267 → 782.910
⚠️ agent_rpm_arm64 (per-PR threshold) +10.85 MiB (1.47% increase, -73.25% of buffer) 740.392 → 751.246 → 755.210
⚠️ agent_suse_amd64 (per-PR threshold) +11.81 MiB (1.54% increase, -81.71% of buffer) 768.455 → 780.267 → 782.910
⚠️ agent_suse_arm64 (per-PR threshold) +10.85 MiB (1.47% increase, -73.25% of buffer) 740.392 → 751.246 → 755.210
⚠️ docker_agent_amd64 (per-PR threshold) +11.79 MiB (1.42% increase, -92.92% of buffer) 827.435 → 839.222 → 840.120
⚠️ docker_agent_arm64 (per-PR threshold) +10.85 MiB (1.32% increase, -73.08% of buffer) 824.427 → 835.281 → 839.280
⚠️ docker_agent_jmx_amd64 (per-PR threshold) +11.79 MiB (1.16% increase, -90.79% of buffer) 1018.347 → 1030.135 → 1031.330
⚠️ docker_agent_jmx_arm64 (per-PR threshold) +10.85 MiB (1.08% increase, -73.09% of buffer) 1004.119 → 1014.973 → 1018.970
Gate failure full details
Quality gate Error type Error message
agent_deb_amd64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 11.81 MiB exceeds the per-PR threshold of 600.0 KiB
agent_rpm_amd64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 11.81 MiB exceeds the per-PR threshold of 600.0 KiB
agent_rpm_arm64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 10.85 MiB exceeds the per-PR threshold of 600.0 KiB
agent_suse_amd64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 11.81 MiB exceeds the per-PR threshold of 600.0 KiB
agent_suse_arm64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 10.85 MiB exceeds the per-PR threshold of 600.0 KiB
docker_agent_amd64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 11.79 MiB exceeds the per-PR threshold of 600.0 KiB
docker_agent_arm64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 10.85 MiB exceeds the per-PR threshold of 600.0 KiB
docker_agent_jmx_amd64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 11.79 MiB exceeds the per-PR threshold of 600.0 KiB
docker_agent_jmx_arm64 PerPRThresholdExceeded (non-blocking: exception granted by @chouquette) On-disk size increase of 10.85 MiB exceeds the per-PR threshold of 600.0 KiB

Note: Some gates exceeded limits but are non-blocking because the size hasn't increased from the ancestor commit.

Successful checks

Info

Quality gate Change Size (prev → curr → max)
agent_deb_amd64_fips +37.16 KiB (0.01% increase, -1.22% of buffer) 715.052 → 715.088 → 718.020
agent_heroku_amd64 +33.16 KiB (0.01% increase, -0.51% of buffer) 316.120 → 316.152 → 322.470
agent_msi +516.27 KiB (0.08% increase, -3.33% of buffer) 647.743 → 648.247 → 662.870
agent_rpm_amd64_fips +37.16 KiB (0.01% increase, -1.22% of buffer) 715.035 → 715.072 → 718.020
agent_rpm_arm64_fips +37.16 KiB (0.01% increase, -0.83% of buffer) 691.525 → 691.561 → 695.900
agent_suse_amd64_fips +37.16 KiB (0.01% increase, -1.22% of buffer) 715.035 → 715.072 → 718.020
agent_suse_arm64_fips +37.16 KiB (0.01% increase, -0.83% of buffer) 691.525 → 691.561 → 695.900
docker_cluster_agent_amd64 +40.0 KiB (0.02% increase, -3.85% of buffer) 212.686 → 212.726 → 213.700
docker_cluster_agent_arm64 +64.0 KiB (0.03% increase, -6.63% of buffer) 225.718 → 225.781 → 226.660
docker_dogstatsd_arm64 +64.0 KiB (0.17% increase, -6.53% of buffer) 37.623 → 37.686 → 38.580
docker_host_profiler_amd64 +4.82 KiB (0.00% increase, -0.04% of buffer) 307.574 → 307.579 → 319.490
iot_agent_deb_amd64 +8.0 KiB (0.02% increase, -0.87% of buffer) 46.651 → 46.659 → 47.550
iot_agent_deb_arm64 +4.0 KiB (0.01% increase, -0.42% of buffer) 43.280 → 43.284 → 44.220
iot_agent_deb_armhf +4.0 KiB (0.01% increase, -0.42% of buffer) 44.094 → 44.097 → 45.020
iot_agent_rpm_amd64 +8.0 KiB (0.02% increase, -0.87% of buffer) 46.651 → 46.659 → 47.550
iot_agent_suse_amd64 +8.0 KiB (0.02% increase, -0.87% of buffer) 46.650 → 46.658 → 47.550
8 successful checks with minimal change (< 2 KiB)
Quality gate Current Size
docker_cws_instrumentation_amd64 7.439 MiB
docker_cws_instrumentation_arm64 6.877 MiB
docker_dogstatsd_amd64 39.557 MiB
docker_host_profiler_arm64 318.793 MiB
dogstatsd_deb_amd64 30.303 MiB
dogstatsd_deb_arm64 28.315 MiB
dogstatsd_rpm_amd64 30.303 MiB
dogstatsd_suse_amd64 30.303 MiB

@dd-octo-sts dd-octo-sts Bot added the internal Identify a non-fork PR label Sep 2, 2026
@github-actions github-actions Bot added the long review PR is complex, plan time to review it label Sep 2, 2026
@astuyve
astuyve requested a review from aiuto September 3, 2026 15:42
@astuyve astuyve added the qa/done QA done before merge and regressions are covered by tests label Sep 3, 2026
@astuyve astuyve added this to the 7.84.0 milestone Sep 3, 2026
…-helper

# Conflicts:
#	go.sum
#	omnibus/config/software/datadog-agent.rb
#	pkg/fleet/installer/packages/datadog_agent_linux.go
#	test/static/static_quality_gates.yml
@astuyve
astuyve requested review from a team and cmourot as code owners September 3, 2026 16:45
@dd-octo-sts dd-octo-sts Bot added the team/container-platform The Container Platform Team label Sep 3, 2026

@aiuto aiuto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will help if you add Bazel tooling directly into https://github.com/DataDog/rshell/tree/main/cmd/rshell
Then we can depend on the BUILD file there directly.

Comment thread tasks/rshell.py Outdated
Comment thread test/static/static_quality_gates.yml Outdated
Comment thread pkg/privateactionrunner/rshell/deps.go
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit aedb15d into main Sep 4, 2026
611 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the rfc/privileged-rshell-helper branch September 4, 2026 22:57
@github-actions github-actions Bot modified the milestones: 7.84.0, 7.85.0 Sep 4, 2026
@astuyve astuyve added the backport/7.84.x Automatically create a backport PR to the 7.84.x branch once the PR is merged label Sep 5, 2026
@temporal-github-worker-1

Copy link
Copy Markdown
Contributor

Backport evaluation: View Slack thread

@dd-octo-sts

dd-octo-sts Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

⚠️ Automatic backport to 7.84.x failed.

This usually happens when the cherry-pick has merge conflicts and needs manual resolution.

To backport manually, run:

git fetch
git worktree add .worktrees/backport-7.84.x 7.84.x
cd .worktrees/backport-7.84.x
git switch --create backport-55731-to-7.84.x
git cherry-pick -x --mainline 1 aedb15d47cb61c56341d8ca209d6a30f9c707286
git push --set-upstream origin backport-55731-to-7.84.x

Workflow logs: https://github.com/DataDog/datadog-agent/actions/runs/33931844443

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport/7.84.x Automatically create a backport PR to the 7.84.x branch once the PR is merged internal Identify a non-fork PR long review PR is complex, plan time to review it qa/done QA done before merge and regressions are covered by tests team/action-platform team/agent-build team/agent-devx team/container-platform The Container Platform Team team/data-streams-monitoring team/ebpf-platform team/fleet-automation team/fleet-remediation team/network-device-monitoring-core team/remote-config team/windows-products

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants