Let's Encrypt support

[Neustradamus]

Dear @FOGProject team,

It is possible to add the Let's Encrypt support?

• https://letsencrypt.org/

Thanks in advance.

[mastacontrola]

If we force letsencrypt, you're effectively asking that all fog installations have a publically accessible domain?

[darksidemilk]

Well first we'd need to adjust the PKI stuff so that you can use your own CA. It is possible but might cause issues with the fog client. The cert renewing every 90 days would mean needing to redo the certs on the client every 90 days and they might lose authentication. I tried doing a lets encrypt cert on a different product's client that also uses certs in a similar way and the constant renew of the cert was a problem. So we would maybe have to adjust the client authentication mechanism to use a different cert or to have it re-obtain the cert and regenerate the token every so often or something like that.
I think it's a good idea, but it's a bit more work than it seems to implement in all certificate related things.
Fog uses certs in a few places

• Client for authentication
• Client for communication (uses cert configured at web server level)
• Web server (can be manually configured to be a different cert than the generated cert)
• ipxe (uses the web server cert by accessing boot file over https)

When you enable https in the installer, ipxe is built with the generated cert as trusted. ipxe may already trust the Let's Encrypt CA out of the box, so the web server and the ipxe might be able to work if you manually change your server to use Let's Encrypt, but if the cert available somewhere in /var/www/fog that gets downloaded by the client is that same cert, it could be an issue.

I have tentative plans to work on providing more flexible options to the PKI of Fog, but I don't know when I'll have time to dedicate to it. There is an old pull request (#354) with some work done on 1.5.x that I need to test out and see if it still works.

Long story short, if we can make the configuration of the PKI config more flexible, allowing Let's Encrypt is possible. I don't know if I'd go as far as an installer option to set it up as that might be a lot to maintain for all environments and there's additional infrastructure required, but having a doc in the docs site could be done. Or having something in fog-community-scripts for it might be something too.

[osiktech]

@mastacontrola: there is a way of setting up a self hosted PKI using smallstep. I am using this to give internal only services the ability to create/renew certs. the beauty is, it supports the LetsEncrypt ACME calls, so it is very easy to setup acme.sh to create/renew certs against the self hosted CA.

[darksidemilk]

That tool looks very intriguing. If you have any examples or want to help with getting that tool integrated into Fog natively or through a plugin that would be awesome @osiktech

[osiktech]

I can provide some insights, but as you need to run a smallstep server to run your internal PKI I would not integrate it, but point to a howto with more explanation.

Also there is another option: One could create an letsencrypt cert using the DNS-Api of certbot or acme.sh. This all depends on how the ssl certs are create within fog. This way one does not need to expose fog externally.

Maybe an option in the installer, which lets the admin decide which ssl certificates to use could be an option? Or even simpler, point to a howto in installer, which explains how to replace the automatic generated certs and what steps are necessary to rebuild ipxe e.g.

[darksidemilk]

I was thinking how to in docs.fogproject.org is going to be the way to go.

[osiktech]

I am in the process of testing this at moment. my findings can be found in this feature request: #794

[mastacontrola]

Picking this back up and tying it to the work that's since landed.

The thread already converged on the right conclusion (thanks @osiktech, @darksidemilk): "Let's Encrypt support" for FOG really means making the PKI flexible enough to use your own CA, because FOG doesn't just terminate TLS at the web server — the fog-client pins the CA and validates the server's chain against it, and iPXE is compiled to trust the CA. So a bare LE cert dropped on the vhost breaks client auth.

That underlying flexibility was implemented under #794 (working-1.6): the installer now supports --external-ca --ca-cert --ca-key --ca-root to sign FOG's server certificate with your own intermediate CA. The intermediate is what gets pinned (ca.cert.der) and the root+intermediate chain is fed to iPXE and the web server.

On top of that, I've written a howto covering both the internal-ACME path and public LE: docs/EXTERNAL_CA_AND_LETSENCRYPT.md.

Practical guidance from that work:

• Recommended: run an internal ACME CA (step-ca / smallstep). Install FOG with --external-ca pointed at step-ca's intermediate, then renew the leaf via acme.sh/certbot against step-ca. Because clients pin the intermediate, leaf renewals are transparent — no client churn, and nothing needs to be publicly exposed.
• Public Let's Encrypt is possible (use DNS-01 so you don't expose the server) but fragile: LE only hands you leaf certs (not a CA key), and LE rotates its intermediates — when the pinned intermediate changes, every client must re-pin. That's the core mismatch with FOG's model.
• No auto-renew today. Cert setup is install-time. Leaf renewal under a stable pinned CA just needs a web-server reload; a CA change requires re-running the installer + clients re-pinning + iPXE rebuild. Robust root-anchored intermediate handling in the client is tracked separately (Add intermediate/root-CA chain support to client trust validation (IsFromCA) — Tier 2 of fogproject#794 zazzles#47).

Given all that, I'd treat this as largely addressed by #794 + the new howto rather than a separate "force Let's Encrypt" feature. Closing as superseded by #794 seems reasonable unless we want to track installer-level ACME auto-renewal as its own enhancement.

[mastacontrola]

Closing as superseded by #794, per the conclusion reached above — #794 is now closed, so this can follow it.

Recapping what exists now, for anyone arriving here from a search:

Use your own CA: installfog.sh --external-ca --web-ca-cert <intermediate.pem> --web-ca-key <intermediate.key> --web-ca-root <root.pem>. The intermediate is what gets pinned as ca.cert.der, and the root+intermediate chain goes to iPXE's TRUST= and the web server. validateExternalCA() checks the intermediate really is a CA and really is signed by the root you supplied, at install time.

Leaf renewal is scripted now. /opt/fog/pki/renewal-helper --zone web re-issues the web leaf under the existing CA and reloads the web server. It deliberately refuses when the certificate resolves outside FOG's own PKI — an ACME-managed leaf is your ACME client's job, and FOG re-issuing over it is how you get a cert/key mismatch that takes the web server down. That check is derived from where the file actually is, not from a stored flag, so it cannot disagree with reality.

New in 1.6: FOG Configuration → Certificates. Shows the chain and expiry dates, downloads each public certificate, takes a root CA upload straight into the server's OS trust store, and carries a "the web certificate chains to a public CA" switch — which is the setting that tells FOG not to rebuild iPXE, because a public root is already cross-certified at boot.

Written up end to end: the Let's Encrypt with FOG how-to, and External CA & Let's Encrypt certificates.

The recommendation from the thread stands, and it is worth restating because it is the non-obvious part: an internal ACME CA (step-ca) beats public Let's Encrypt for FOG. Clients pin the intermediate, so leaf renewals are transparent — whereas LE rotates its intermediates, and every rotation would make every client re-pin. Thanks @osiktech and @darksidemilk for getting the thread to that conclusion.

Still not done, and deliberately out of scope here: installer-level ACME auto-renewal, and root-anchored intermediate handling in the client (FOGProject/zazzles#47). If installer-driven ACME is wanted, it deserves its own issue rather than living on the end of this one.