Skip to content

Fix Pulumi deployment due to a partial apply#6

Merged
cdunster merged 11 commits intomainfrom
fix-pulumi-deployment
Mar 26, 2026
Merged

Fix Pulumi deployment due to a partial apply#6
cdunster merged 11 commits intomainfrom
fix-pulumi-deployment

Conversation

@cdunster
Copy link
Copy Markdown
Contributor

@cdunster cdunster commented Mar 26, 2026

After a partial/failed apply then some resources are not re-created when the droplet changes meaning that some files are missing from the droplet. I've added the droplet ID as a trigger to all resources so if the droplet changes then all resources are recreated causing all the commands and copies to run again.

All the Nomad variables were missing or out-of-date too so I've updated them as the new server will need them.

Summary by CodeRabbit

  • Refactor
    • Remote setup steps now reliably re-run when a target instance changes; dependency wiring simplified for directory, certificate, ownership, and service sequencing.
  • Configuration
    • Updated InfluxDB token and added a durable-objects URL and secret for the Nomad server.

There was an issue where a partial apply caused these resources to not
be recreated meaning the contents were not copied to or created on the
remote machine.
@cdunster cdunster self-assigned this Mar 26, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 26, 2026

Warning

Rate limit exceeded

@cdunster has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 14 minutes and 44 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9ce145a9-0027-40df-8960-0d5188924bda

📥 Commits

Reviewing files that changed from the base of the PR and between 571839b and ed1921b.

📒 Files selected for processing (2)
  • Pulumi.nomad-server.yaml
  • main.go

Walkthrough

Updated trigger/dependency wiring in main.go: many remote copy/command resources now include droplet.ID() in their Triggers, several chown commands were renamed and had DependsOn simplified, server cert creation and ACL/policy/vars steps had trigger/dependency changes. Pulumi.nomad-server.yaml adds UNYT secret/URL and replaces the Influx token.

Changes

Cohort / File(s) Summary
Remote trigger & dependency updates
main.go
Added droplet.ID() to many NewCommand/NewCopyToRemote Triggers; renamed chownEtcNomadDirPreCert/chownEtcNomadDirFinal; simplified/removed several DependsOn links; server cert creation, print, ACL bootstrap/apply and service restart triggers adjusted to use resource IDs and/or droplet.ID().
Directory/flow adjustments
main.go
create-opt-nomad-data-dir no longer depends on reservedIpAssign (now depends only on waitForNomadUser); pre-cert chown dependencies tightened; server cert creation now removes CA key file and depends on pre-cert chown.
Vars / secrets wiring
main.go, Pulumi.nomad-server.yaml
Renamed add-influx-db-token-varadd-nomad-jobs-vars; expanded vars step to require unytDurableObjectsURL and unytDurableObjectsSecret; updated nomad var put target to nomad/jobs and include UNYT env vars. In Pulumi file: replaced influxDBToken.secure value and added unytDurableObjectsSecret.secure and unytDurableObjectsURL.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • Feat/improve reliability #5 — Similar adjustments to remote copy/command trigger and dependency wiring affecting Nomad server provisioning.

Suggested reviewers

  • ThetaSinner
  • jost-s
🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Fix Pulumi deployment due to a partial apply' clearly describes the main change: adding droplet ID triggers to remote resources to prevent missing files after partial Pulumi applies.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-pulumi-deployment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@holochain-release-automation2
Copy link
Copy Markdown
Collaborator

holochain-release-automation2 commented Mar 26, 2026

🍹 preview on nomad-server/holochain/nomad-server

Pulumi report

View in Pulumi Cloud

  Previewing update (holochain/nomad-server)

View Live: https://app.pulumi.com/holochain/nomad-server/nomad-server/previews/c83979f5-d24b-4924-88f5-56ef56d19f87

pulumi:pulumi:Stack: (same)
  [urn=urn:pulumi:nomad-server::nomad-server::pulumi:pulumi:Stack::nomad-server-nomad-server]
  +-command:remote:Command: (replace)
      [id=create-etc-nomad-dird430c974]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::create-etc-nomad-dir]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    + triggers: [
    +     [0]: "560913518"
      ]
  +-command:remote:CopyToRemote: (replace)
      [id=201dcf7f]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:CopyToRemote::copy-nomad-service-config]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    ~ triggers: [
        + [1]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=wait-for-nomad-user34e53e3e]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::wait-for-nomad-user]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    + triggers: [
    +     [0]: "560913518"
      ]
  +-command:remote:CopyToRemote: (replace)
      [id=f22dfbd1]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:CopyToRemote::copy-nomad-config]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - source  : asset(file:6775693) { ./nomad.hcl }
    + source  : asset(file:b2202f8) { ./nomad.hcl }
    ~ triggers: [
        - [0]: asset(file:6775693) { ./nomad.hcl }
        + [0]: asset(file:b2202f8) { ./nomad.hcl }
        + [1]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=copy-ca-keye6c21aca]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::copy-ca-key]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    ~ triggers: [
        + [1]: "560913518"
      ]
  +-command:remote:CopyToRemote: (replace)
      [id=2c5d175f]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:CopyToRemote::copy-job-runner-policy]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    ~ triggers: [
        + [1]: "560913518"
      ]
  +-command:remote:CopyToRemote: (replace)
      [id=b0892fa4]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:CopyToRemote::copy-ca-cert]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    ~ triggers: [
        + [1]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=create-opt-nomad-data-dir3cddce2d]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::create-opt-nomad-data-dir]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    + triggers: [
    +     [0]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=chown-etc-nomad-dir-before-server-cert53de8fbf]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::chown-etc-nomad-dir-before-server-cert]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: [unknown]
    +     [1]: "copy-ca-keyed37bcad"
    +     [2]: [unknown]
    +     [3]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=create-server-cert33b81701]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::create-server-cert]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    ~ create  : "cd /etc/nomad.d && rm -f global-server-nomad*.pem && nomad tls cert create -server -additional-dnsname=nomad-server-01.holochain.org" => "cd /etc/nomad.d && rm -f global-server-nomad*.pem && nomad tls cert create -server -additional-dnsname=nomad-server-01.holochain.org && rm -f nomad-agent-ca-key.pem"
    - triggers: [secret]
    + triggers: [
    +     [0]: [unknown]
    +     [1]: "copy-ca-keyed37bcad"
    +     [2]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=chown-etc-nomad-dirb49d03f2]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::chown-etc-nomad-dir]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: "create-server-cert6e34acd8"
    +     [1]: [unknown]
    +     [2]: [unknown]
    +     [3]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=print-server-cert29133052]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::print-server-cert]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: "create-server-cert6e34acd8"
    +     [1]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=enable-nomad-service6d825c44]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::enable-nomad-service]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    + triggers: [
    +     [0]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=start-nomad-servicefa6450bd]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::start-nomad-service]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    ~ triggers: [
        - [0]: asset(file:6775693) { ./nomad.hcl }
        + [0]: [unknown]
        - [1]: asset(file:51f7bea) { ./nomad.service }
        + [1]: [unknown]
        + [2]: [unknown]
        + [3]: "create-server-cert6e34acd8"
        + [4]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=acl-bootstrapb636019b]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::acl-bootstrap]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    ~ triggers: [
        + [1]: "560913518"
      ]
  + command:remote:Command: (create)
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::add-nomad-jobs-vars]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
      addPreviousOutputInEnv: true
      connection            : [secret]
      create                : "nomad var put -address=https://localhost:4646 -ca-cert=/etc/nomad.d/nomad-agent-ca.pem -token=\"$LC_ACL_TOKEN\" nomad/jobs INFLUX_TOKEN=\"$LC_INFLUX_TOKE..."
      environment           : {
          LC_ACL_TOKEN                  : [secret]
          LC_INFLUX_TOKEN               : [secret]
          LC_UNYT_DURABLE_OBJECTS_SECRET: [secret]
          LC_UNYT_DURABLE_OBJECTS_URL   : "https://wind-tunnel-durable-objects.holochain.org/"
      }
      triggers              : [
          [0]: [secret]
          [1]: "560913518"
      ]
  +-command:remote:Command: (replace)
      [id=apply-job-runner-policyfc2c752e]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::apply-job-runner-policy]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: [unknown]
    +     [1]: "acl-bootstrap2e155df9"
    +     [2]: "560913518"
      ]
  - command:remote:Command: (delete)
      [id=add-influx-db-token-varea4f53ae]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::add-influx-db-token-var]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
      addPreviousOutputInEnv: true
      connection            : [secret]
      create                : "nomad var put -address=https://localhost:4646 -ca-cert=/etc/nomad.d/nomad-agent-ca.pem -token=\"$LC_ACL_TOKEN\" nomad/jobs/run_scenario INFLUX_TOKEN=\"$L..."
      environment           : {
          LC_ACL_TOKEN   : [secret]
          LC_INFLUX_TOKEN: [secret]
      }
      triggers              : [
          [0]: [secret]
      ]
Resources:
  + 1 to create
  - 1 to delete
  +-16 to replace
  18 changes. 5 unchanged
  

@pulumi
Copy link
Copy Markdown

pulumi bot commented Mar 26, 2026

🍹 The Update (preview) for holochain/nomad-server/nomad-server (at ed1921b) was successful.

✨ Neo Explanation

A change to the Nomad server configuration file is triggering a full re-provisioning of the Nomad server setup, including service restarts and ACL re-bootstrapping, alongside a refactor of how job variables are injected into the server environment.

Root Cause Analysis

The Nomad configuration file (copy-nomad-config) was modified — the [diff: ~source,triggers] on that resource indicates the source content changed. Additionally, the add-influx-db-token-var command is being replaced by a new add-nomad-jobs-vars command, suggesting that how environment/job variables are configured on the Nomad server has been refactored (likely consolidating or renaming the variable setup step).

Dependency Chain

The updated Nomad config file is the root trigger. Because nearly all remote commands use triggers linked to this config, changing it causes a full cascade: all file copy operations and remote commands (directory creation, cert setup, service management, ACL bootstrapping, policy application) are marked for replacement. Separately, add-influx-db-token-var is deleted and replaced with a new add-nomad-jobs-vars command, reflecting a rename/refactor of that setup step.

Risk analysis

Medium risk. The entire Nomad server setup sequence will be torn down and re-run — including service restarts (start-nomad-service, enable-nomad-service) and ACL bootstrapping (acl-bootstrap). This means the Nomad server will experience a restart and brief downtime during the apply. The ACL bootstrap re-run could be sensitive if tokens are regenerated. No stateful storage resources (databases, volumes) are being replaced.

Resource Changes

    Name                                    Type                         Operation
+-  create-etc-nomad-dir                    command:remote:Command       replaced
+-  copy-nomad-config                       command:remote:CopyToRemote  replaced
+-  apply-job-runner-policy                 command:remote:Command       replaced
+-  copy-ca-key                             command:remote:Command       replaced
+-  copy-job-runner-policy                  command:remote:CopyToRemote  replaced
+-  copy-ca-cert                            command:remote:CopyToRemote  replaced
+-  create-server-cert                      command:remote:Command       replaced
+-  chown-etc-nomad-dir                     command:remote:Command       replaced
+-  enable-nomad-service                    command:remote:Command       replaced
+-  start-nomad-service                     command:remote:Command       replaced
+-  copy-nomad-service-config               command:remote:CopyToRemote  replaced
+-  wait-for-nomad-user                     command:remote:Command       replaced
+   add-nomad-jobs-vars                     command:remote:Command       create
-   add-influx-db-token-var                 command:remote:Command       delete
+-  create-opt-nomad-data-dir               command:remote:Command       replaced
+-  chown-etc-nomad-dir-before-server-cert  command:remote:Command       replaced
+-  print-server-cert                       command:remote:Command       replaced
+-  acl-bootstrap                           command:remote:Command       replaced

These resources already use the reserved IP in the conn so its not
needed.
coderabbitai[bot]

This comment was marked as outdated.

coderabbitai[bot]

This comment was marked as outdated.

@cdunster cdunster requested a review from a team March 26, 2026 14:36
@holochain holochain deleted a comment from cocogitto-bot bot Mar 26, 2026
@cdunster cdunster force-pushed the fix-pulumi-deployment branch from 407e446 to 571839b Compare March 26, 2026 15:40
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@main.go`:
- Around line 276-289: The Create command string in the remote.NewCommand call
is currently passing the literal text "LC_UNYT_DURABLE_OBJECTS_SECRET" instead
of expanding the environment variable; update the Create string in the
CommandArgs passed to remote.NewCommand so UNYT_DURABLE_OBJECTS_SECRET is set to
the environment variable (i.e., reference $LC_UNYT_DURABLE_OBJECTS_SECRET or
\"$LC_UNYT_DURABLE_OBJECTS_SECRET\" consistent with the other vars) so the
secret stored by nomad var put comes from unytDurableObjectsSecret rather than a
literal.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2dd70cce-b966-4cd2-8d59-f58a04f238c0

📥 Commits

Reviewing files that changed from the base of the PR and between 5fdbc22 and 571839b.

📒 Files selected for processing (2)
  • Pulumi.nomad-server.yaml
  • main.go

mattyg
mattyg previously approved these changes Mar 26, 2026
@cocogitto-bot
Copy link
Copy Markdown

cocogitto-bot bot commented Mar 26, 2026

✔️ fddaddb...ed1921b - Conventional commits check succeeded.

@cdunster cdunster merged commit 2b68cb9 into main Mar 26, 2026
6 checks passed
@cdunster cdunster deleted the fix-pulumi-deployment branch March 26, 2026 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants