feat: create single Nomad server running on DigitalOcean droplet by cdunster · Pull Request #3 · holochain/nomad-server

cdunster · 2025-04-04T15:38:21Z

This is the Pulumi definition for the Nomad Server that is required for #holochain/wind-tunnel/issues/132 and can be found at https://nomad-server-01.holochain.org:4646/ui.

Before this, changing the droplet replaced the reserved IP address which changed the actual address.

It's not working so just remove it for now.

holochain-release-automation2 · 2025-04-04T15:39:43Z

🍹 `preview` on nomad-server/holochain/nomad-server

Pulumi report

View in Pulumi Cloud

  Previewing update (holochain/nomad-server)

View Live: https://app.pulumi.com/holochain/nomad-server/nomad-server/previews/b1f3a3ca-5570-488c-8047-d17d0b61afac

pulumi:pulumi:Stack: (same)
  [urn=urn:pulumi:nomad-server::nomad-server::pulumi:pulumi:Stack::nomad-server-nomad-server]
  +-command:remote:Command: (replace)
      [id=chown-etc-nomad-dir-before-server-certc71d31ab]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::chown-etc-nomad-dir-before-server-cert]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: asset(file:f687a96) { ./nomad-agent-ca.pem }
    +     [1]: [secret]
    +     [2]: asset(file:6775693) { ./nomad.hcl }
      ]
  +-command:remote:Command: (replace)
      [id=create-server-cert1ba48512]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::create-server-cert]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: asset(file:f687a96) { ./nomad-agent-ca.pem }
    +     [1]: [secret]
      ]
  +-command:remote:Command: (replace)
      [id=print-server-cert13df651b]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::print-server-cert]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: asset(file:f687a96) { ./nomad-agent-ca.pem }
    +     [1]: [secret]
      ]
  +-command:remote:Command: (replace)
      [id=chown-etc-nomad-dir52b48959]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::chown-etc-nomad-dir]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: asset(file:f687a96) { ./nomad-agent-ca.pem }
    +     [1]: [secret]
    +     [2]: asset(file:6775693) { ./nomad.hcl }
    +     [3]: asset(file:1289f09) { ./job-runner.policy.hcl }
      ]
  +-command:remote:Command: (replace)
      [id=apply-job-runner-policy8353b57a]
      [urn=urn:pulumi:nomad-server::nomad-server::command:remote:Command::apply-job-runner-policy]
      [provider=urn:pulumi:nomad-server::nomad-server::pulumi:providers:command::default_1_0_2::3d5f4907-5ccd-4d31-8098-9549b40692ec]
    - triggers: [secret]
    + triggers: [
    +     [0]: asset(file:1289f09) { ./job-runner.policy.hcl }
    +     [1]: [secret]
      ]
Resources:
  +-5 to replace
  17 unchanged

main.go

This will be handled by Pulumi SSH commands. BREAKING CHANGE: Will re-create the droplet.

Nomad was trying to read the policy as a config.

Sometimes the droplet was created without this access key that is needed later.

Pulumi.nomad-server.yaml

cdunster · 2025-04-09T09:38:30Z

@mattyg, @jost-s, and @ThetaSinner, I've added quite a lot to the Pulumi code since your last reviews. Now, Pulumi performs the Nomad setup for us automatically.

jost-s

I'm too late with my review, but anyway. I wanted to run the commands today, so here is my text from yesterday.

You went to town on the go coding there!
I can follow the description of all the certificates and tokens. I trust that it works, I haven't run the commands myself.

jost-s · 2025-04-09T19:39:14Z

README.md

+
+## Changing the TLS certificate authority certificate
+
+Nomad uses TLS for encrypted communications with the server(s) for this to work


Suggested change

Nomad uses TLS for encrypted communications with the server(s) for this to work

Nomad uses TLS for encrypted communications with the server(s). For this to work

jost-s · 2025-04-09T19:40:57Z

README.md

+Some changes to the definition of the DigitalOcean droplet will require the
+droplet to be destroyed and re-created.
+
+If this happens, or if we want to deliberately re-create the server(s), then


Suggested change

If this happens, or if we want to deliberately re-create the server(s), then

If this happens, or if we want to deliberately re-create the server(s),

cdunster · 2025-04-10T13:56:43Z

Ah, sorry @jost-s. I thought that it seems to be working, and we can always make changes later in separate PRs, so I just went for it 😁

You went to town on the go coding there!

Yeah, once I found the command to run arbitrary commands and copy files via SSH then I saw no reason to not just automate everything

jost-s · 2025-04-10T20:37:22Z

Totally agree. I'll come back to this if necessary once I've tried out these steps.

cdunster added 18 commits February 5, 2025 19:12

feat: add hashicorp apt repo for latest nomad version

9f6daa2

feat: write nomad config to default location

809448a

feat: start nomad as a service

3e4f2c8

feat: run nomad server as nomad user

ffe1398

feat: enable ACL to restrict access to Nomad server

87138bc

feat: advertise public IPv4 address on server

73b7f83

feat: create reserved IPv4 for nomad server

899e5c4

chore: add nomad package

75959f0

chore: add nomad client config for testing

469f9dc

feat: untie reserved IP from droplet lifecycle

9668ae7

Before this, changing the droplet replaced the reserved IP address which changed the actual address.

refactor: don't unnecessarily convert droplet size to string

af968d5

feat: don't advertise serf via public IP

1b81cad

feat: enable TLS and create certs on boot

dc6c708

docs: add section in README for instructions to deploy a server

90149fb

feat: allow clients to connect without TLS

e3fe3fc

feat: add nomad-server-01.holochain.org as DNS name for certificate

45d269d

feat: remove broken verifying server hostname option

48e095c

It's not working so just remove it for now.

feat: increase job lifetime to 24 hours

e5373e2

cdunster requested a review from a team April 4, 2025 15:38

cdunster self-assigned this Apr 4, 2025

ThetaSinner reviewed Apr 7, 2025

View reviewed changes

main.go Show resolved Hide resolved

ThetaSinner previously approved these changes Apr 7, 2025

View reviewed changes

cdunster added 7 commits April 7, 2025 17:54

docs: fix typo in README

6325427

chore: update go packages

21b0bae

feat: add ssh key for this workflow to DigitalOcean

6965d42

feat!: remove nomad setup via cloud-init

65c9087

This will be handled by Pulumi SSH commands. BREAKING CHANGE: Will re-create the droplet.

fix: make reserved IP assignment depend on droplet and reserved IP

0272f9d

feat: copy existing CA cert

b1ed71f

feat: copy nomad.service file

19df92c

cdunster added 7 commits April 8, 2025 19:13

fix: hide stdout when bootstrapping ACL

6ade18c

feat: reduce ssh timeout from infinite to 60

e322a23

fix: use single nomad config file instead of dir

186488b

Nomad was trying to read the policy as a config.

fix: add dependency on creating SSH key for getting SSH keys

2bb9879

Sometimes the droplet was created without this access key that is needed later.

feat: update the nomad service start command

55f88ba

chore: update all certs and keys after stack delete

94a6ab7

fix: use bootstrap token to add job-runner policy

bf65d61

cdunster force-pushed the single-nomad-server-running-on-droplet branch from fa95579 to bf65d61 Compare April 8, 2025 17:13

fix: correct Triggers as they don't work on previous jobs

529bbde

mattyg reviewed Apr 8, 2025

View reviewed changes

Pulumi.nomad-server.yaml Show resolved Hide resolved

cdunster added 2 commits April 9, 2025 10:38

chore: update config after moving stack

04e0133

fix: copy policy file depends on etc/nomad.d dir

01418b0

cdunster force-pushed the single-nomad-server-running-on-droplet branch from 7d2192d to 01418b0 Compare April 9, 2025 08:53

cdunster added 2 commits April 9, 2025 10:58

docs: fix README format issues

52f958a

docs: update the README

d6eb769

cdunster requested review from ThetaSinner, jost-s and mattyg April 9, 2025 09:36

cdunster added 5 commits April 9, 2025 17:28

feat: add InfluxDB token to server

b8d1b42

docs: add section about replacing the InfluxDB token

e5ff508

feat: add step to print the new server certificate

c641495

docs: fix typo in README

d7674de

docs: add extra info about resetting the ACL token in README

3736b8f

mattyg approved these changes Apr 9, 2025

View reviewed changes

cdunster merged commit 0e817dc into main Apr 10, 2025
3 checks passed

cdunster deleted the single-nomad-server-running-on-droplet branch April 10, 2025 11:58

jost-s reviewed Apr 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: create single Nomad server running on DigitalOcean droplet#3

feat: create single Nomad server running on DigitalOcean droplet#3
cdunster merged 54 commits intomainfrom
single-nomad-server-running-on-droplet

cdunster commented Apr 4, 2025 •

edited

Loading

Uh oh!

holochain-release-automation2 commented Apr 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cdunster commented Apr 9, 2025

Uh oh!

Uh oh!

jost-s left a comment

Uh oh!

jost-s Apr 9, 2025

Uh oh!

jost-s Apr 9, 2025

Uh oh!

cdunster commented Apr 10, 2025

Uh oh!

jost-s commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		## Changing the TLS certificate authority certificate

		Nomad uses TLS for encrypted communications with the server(s) for this to work

	If this happens, or if we want to deliberately re-create the server(s), then
	If this happens, or if we want to deliberately re-create the server(s),

Conversation

cdunster commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holochain-release-automation2 commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🍹 preview on nomad-server/holochain/nomad-server

Uh oh!

Uh oh!

Uh oh!

cdunster commented Apr 9, 2025

Uh oh!

Uh oh!

jost-s left a comment

Choose a reason for hiding this comment

Uh oh!

jost-s Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

jost-s Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

cdunster commented Apr 10, 2025

Uh oh!

jost-s commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cdunster commented Apr 4, 2025 •

edited

Loading

holochain-release-automation2 commented Apr 4, 2025 •

edited

Loading

🍹 `preview` on nomad-server/holochain/nomad-server