Cert for both Wildcard and domain itself

[DarkSuniuM]

Which version of python are you using?

3.8.3

What operating system and version of operating system are you using?

Tested on Archlinux, Alpine and Debian Buster

What version of sewer are you using?

0.8.2

What did you do? (be as detailed as you can)

Registered a cert on a wildcard address, *.mydomain.com, The cert only had *.mydomain.com as domains and didn't work on mydomain.com itself,
Tried to use mydomain.com as domain_name and ['*.mydomain.com'] as domain_alt_names,
resulted in an error which is pasted below.

from sewer.dns_providers import PowerDNSDns as PowerDNS
from sewer.client import Client

provider = PowerDNS(...)
domain_name = 'mydomain.com'
domain_alt_names = ['*.mydomain.com']  # I also swapped domain_name with domain_alt_names, didn't work
c = Client(domain_name=domain_name, provider=provider, domain_alt_names=domain_alt_names, LOG_LEVEL='DEBUG')
cert = c.cert()  # Error happens!

What did you expect to see/happen/not happen?

Get a new cert for the given domain and wildcard

What did you actually see/happen?

Error: Unable to issue certificate. error=Checks done=3. Max checks allowed=3. Interval between checks=8seconds.

Paste here the log output generated by sewer, if any. Please remember to remove any sensitive items from the log before pasting here.

If you can, run sewer with loglevel set to debug; eg sewer --loglevel DEBUG

get_acme_endpoints
get_acme_endpoints_response. status_code=200
create_certificate_key
create_csr
create_account_key
intialise_success, sewer_version=0.8.2, domain_names=['*.my-secret-domain.com', 'my-secret-domain.com'], acme_server=https://acme-v02.api...
get_certificate
acme_register (newAccount)
make_signed_acme_request
get_acme_header
get_nonce
sign_message
acme_register_response. status_code=200. response={'key': {'kty': 'RSA', 'n': '42Lo3x02xut1IUTiG_D4_gNuvxGkT-uzJd_X79BvmQHFpwn0JVuBVjf92EyHXeemW0g5yXb9o79-ZjeSgZds-iHCa1Gv7encTU-J8TAK89hmE_uY7fEKD5_kUMpnxNeJESPdmUg7k9JIwaIcGNtgP8-PKj08-vCE1wNtLCt7GbOuPQ0wWvypPBB3I4e5DqwPMK2ZR_hHqQtN5BVuKbR6dUEk_74mv-tKA0P6Pr3hv0z_NzG020ipYwG6_DD-W5zNMZigr9QCGMOF335pd6DxaWutBCmjW0sOOyiFWUb1JuV6LLj8porRfls9fDKlS9wPNbfDC4v4sRwvEVTUbanoAQ', 'e': 'AQAB'}, 'contact': [], 'initialIp': '159.70.250.34', 'createdAt': '2020-07-18T03:20:30Z', 'status': 'valid'}
acme_register_success
apply_for_cert_issuance (newOrder)
make_signed_acme_request
get_acme_header
get_nonce
sign_message
apply_for_cert_issuance_response. status_code=201. response={'status': 'pending', 'expires': '2020-07-25T03:21:34.644688551Z', 'identifiers': [{'type': 'dns', 'value': '*.my-secret-domain.com'}, {'type': 'dns', 'value': 'my-secret-domain.com'}], 'authorizations': ['https://acme-v02.api.letsencrypt.org/acme/authz-v3/5949220217', 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/5949234659'], 'finalize': 'https://acme-v02.api.letsencrypt.org/acme/finalize/91644537/4256480660'}
apply_for_cert_issuance_success
get_identifier_authorization for https://acme-v02.api.letsencrypt.org/acme/authz-v3/5949220217
make_signed_acme_request
get_acme_header
get_nonce
sign_message
get_identifier_authorization_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'valid', 'expires': '2020-08-17T03:20:56Z', 'challenges': [{'type': 'dns-01', 'status': 'valid', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949220217/ztPs0A', 'token': '_OoBDNIYfyhy6aOModDIkaWI_h6fXXLt0QRfUxTGsbE', 'validationRecord': [{'hostname': 'my-secret-domain.com'}]}]}
get_identifier_authorization_success. identifier_auth={'domain': 'my-secret-domain.com', 'url': 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/5949220217', 'wildcard': None, 'token': '_OoBDNIYfyhy6aOModDIkaWI_h6fXXLt0QRfUxTGsbE', 'challenge_url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949220217/ztPs0A'}
get_identifier_authorization got https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949220217/ztPs0A, token=_OoBDNIYfyhy6aOModDIkaWI_h6fXXLt0QRfUxTGsbE
get_keyauthorization
get_identifier_authorization for https://acme-v02.api.letsencrypt.org/acme/authz-v3/5949234659
make_signed_acme_request
get_acme_header
get_nonce
sign_message
get_identifier_authorization_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'pending', 'expires': '2020-07-25T03:21:34Z', 'challenges': [{'type': 'dns-01', 'status': 'pending', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw', 'token': '1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc'}], 'wildcard': True}
get_identifier_authorization_success. identifier_auth={'domain': 'my-secret-domain.com', 'url': 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/5949234659', 'wildcard': True, 'token': '1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc', 'challenge_url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw'}
get_identifier_authorization got https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw, token=1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc
get_keyauthorization
check_authorization_status
make_signed_acme_request
get_acme_header
get_nonce
sign_message
check_authorization_status_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'valid', 'expires': '2020-08-17T03:20:56Z', 'challenges': [{'type': 'dns-01', 'status': 'valid', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949220217/ztPs0A', 'token': '_OoBDNIYfyhy6aOModDIkaWI_h6fXXLt0QRfUxTGsbE', 'validationRecord': [{'hostname': 'my-secret-domain.com'}]}]}
check_authorization_status_success
check_authorization_status
make_signed_acme_request
get_acme_header
get_nonce
sign_message
check_authorization_status_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'pending', 'expires': '2020-07-25T03:21:34Z', 'challenges': [{'type': 'dns-01', 'status': 'pending', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw', 'token': '1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc'}], 'wildcard': True}
check_authorization_status_success
respond_to_challenge for 1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc.-hJdYNZhhs2-XosyWmDOFK6d2o7BG7xejCeMAmiZLr4 at https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw
make_signed_acme_request
get_acme_header
get_nonce
sign_message
respond_to_challenge_response. status_code=200. response={'type': 'dns-01', 'status': 'pending', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw', 'token': '1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc'}
respond_to_challenge_success
check_authorization_status
make_signed_acme_request
get_acme_header
get_nonce
sign_message
check_authorization_status_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'valid', 'expires': '2020-08-17T03:20:56Z', 'challenges': [{'type': 'dns-01', 'status': 'valid', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949220217/ztPs0A', 'token': '_OoBDNIYfyhy6aOModDIkaWI_h6fXXLt0QRfUxTGsbE', 'validationRecord': [{'hostname': 'my-secret-domain.com'}]}]}
check_authorization_status_success
check_authorization_status
make_signed_acme_request
get_acme_header
get_nonce
sign_message
check_authorization_status_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'invalid', 'expires': '2020-07-25T03:21:34Z', 'challenges': [{'type': 'dns-01', 'status': 'invalid', 'error': {'type': 'urn:ietf:params:acme:error:unauthorized', 'detail': 'Incorrect TXT record "_aj95rmU-P04RSkfKsayUwrx4WkFCpwnS97XiRU3X7Y" found at _acme-challenge.my-secret-domain.com', 'status': 403}, 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw', 'token': '1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc'}], 'wildcard': True}
make_signed_acme_request
get_acme_header
get_nonce
sign_message
check_authorization_status_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'invalid', 'expires': '2020-07-25T03:21:34Z', 'challenges': [{'type': 'dns-01', 'status': 'invalid', 'error': {'type': 'urn:ietf:params:acme:error:unauthorized', 'detail': 'Incorrect TXT record "_aj95rmU-P04RSkfKsayUwrx4WkFCpwnS97XiRU3X7Y" found at _acme-challenge.my-secret-domain.com', 'status': 403}, 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw', 'token': '1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc'}], 'wildcard': True}
make_signed_acme_request
get_acme_header
get_nonce
sign_message
check_authorization_status_response. status_code=200. response={'identifier': {'type': 'dns', 'value': 'my-secret-domain.com'}, 'status': 'invalid', 'expires': '2020-07-25T03:21:34Z', 'challenges': [{'type': 'dns-01', 'status': 'invalid', 'error': {'type': 'urn:ietf:params:acme:error:unauthorized', 'detail': 'Incorrect TXT record "_aj95rmU-P04RSkfKsayUwrx4WkFCpwnS97XiRU3X7Y" found at _acme-challenge.my-secret-domain.com', 'status': 403}, 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/5949234659/V2s3Mw', 'token': '1x1aJiQVv7IAKpIrUDzwkxtzuKF6YC0OjL-dq15ziWc'}], 'wildcard': True}
Error: Unable to issue certificate. error=Checks done=3. Max checks allowed=3. Interval between checks=8seconds.
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/sewer/client.py", line 835, in cert
    return self.get_certificate()
  File "/usr/local/lib/python3.7/site-packages/sewer/client.py", line 777, in get_certificate
    raise e
  File "/usr/local/lib/python3.7/site-packages/sewer/client.py", line 768, in get_certificate
    self.check_authorization_status(chal["auth_url"], ["valid"])
  File "/usr/local/lib/python3.7/site-packages/sewer/client.py", line 537, in check_authorization_status
    self.ACME_AUTH_STATUS_WAIT_PERIOD,
StopIteration: Checks done=3. Max checks allowed=3. Interval between checks=8seconds.

[mmaney]

@DarkSuniuM it loks like it's able to validate one of the two challenges but fails the second. This makes me wonder if PowerDNS is yet another of our legacy DNS drivers that doesn't handle the "duplicate" challenges - two different TXT values have to be attached to one DNS name.

@kylejohnson You started this driver! Any thoughts? I see some discussion in #147, it looks like you confirmed that the domain + *.domain didn't work in the code as merged, but if a fix came up I can't find it.

[DarkSuniuM]

@DarkSuniuM it loks like it's able to validate one of the two challenges but fails the second. This makes me wonder if PowerDNS is yet another of our legacy DNS drivers that doesn't handle the "duplicate" challenges - two different TXT values have to be attached to one DNS name.

@kylejohnson You started this driver! Any thoughts? I see some discussion in #147, it looks like you confirmed that the domain + *.domain didn't work in the code as merged, but if a fix came up I can't find it.

Yeah, At the same time I checked PowerDNS database and also I sent DNS queries to check the TXT record value, I confirm that it only sets up TXT record for the first domain and it does not cleanup the first one, till it finishes the second challenge.

[mmaney]

The current design of sewer's get_certificate() assumes that all challenges can be posted at once, validated by the ACME server, and then taken down. In principle it could handle the publish/validate/remove for each one separately, but that would be more of a rewriting than modifying. I'm not keen to embark on that at this time (if ever, really), especially since in the cases so far brought up it has been possible to work around the issue. Offhand I think route53 and gandi have been fixed recently, eg. Oh, and I have a note that cloudflare has a patch reported to let it work, though I don't think it's been posted here.

... not sure if route53 ever had the wild+bare domain issue - I only see where it was one that got just-the-wildcard fixed. And I don't know, now, why I have it marked as working. :-(

[DarkSuniuM]

Why not just adding a parameter for wildcard?
Since the get_certificate appends all of the domains (domain_name and domain_alt_names) into a single list and iterates over them, With a small change of API, sewer will be able to handle both wildcard and the domain itself without having any problem

# ...
domain_name = ['test.com', True]  # the second element is for the wildcard, I don't like this, but it's a workaround.
# OR
domain_name = 'test.com'  # With a simple type check, we can find out if the domain needs the wild card or not.
# The same logic can be used for domain_alt_names
# ...

Or even having a separated class for Domain, with a wildcard parameter to find out if the *.domain needs to be issued or not

# ...
class Domain:  # Just an example.
    domain_name: str
    wildcard: bool = False

domain = Domain()
domain.domain_name = 'test.com'
domain.wildcard = True
client = Client(domain=domain, provider=DNSProvier) # Instead of `domain_name`, it is easier to use a domain parameter and have the wildcard status on that
# ...

[mmaney]

Why not just add a parameter for wildcard?

I'm glad you asked - this is something I've been meaning to write up for the docs, so let's see what a first draft would look like...

1. The pragmatic issue is that to add ANYTHING to the Legacy DNS interface requires updating all the existing drivers to accept it. While this sort of change shouldn't be too difficult, my inability to actually test the changes against all the varied services makes me leary of doing it if there's another way of achieving the objective.

2. Historically, sewer used to pass a "flag" (prepended "*." on the wildcard challenges) to identify wildcards! After fixing a number of drivers to strip that flag as the bug "cannot create wildcard cert" came around, I surveyed all the Legacy DNS providers and found that none of them used the flag. The wildcard bugs were due to not stripping the flag or to stripping it in the "add TXT" path and not the "remove TXT" path, which would then of course fail. So FIX #162 - removed "*." added by client.py, removed often incomplete … #163 stripped all of that out.

3. In any case the wildcard flag wouldn't fix the issue with PowerDns. Like other services that have shown up with this "wildcard + bare domain" bug, the problem is that the service and/or its API (or the driver's use of the API) doesn't Just Work when there's more than one TXT for a single identity. That problem MUST be addressed in the driver itself.

4. A better way already exists: the new driver interface rooted in ProviderBase in auth.py. This passes all the challenges to the driver at one time and either already does (perhaps in master) or soon will pass a wildcard flag as well as other values from the ACME authz response - not because I actually anticipate most/any of that "extra" material being used in a DNS driver; it's there as a hedge against newer validation methods, should they ever be of interest to sewer.

From what I found in a quick scan of PowerDns' docs, it appears that the problem is probably just in how the driver uses the API. Both the backing store and API are pretty explicit about accepting multiple instances of same (name, type) records. Granted, it's in a manner that might be cumbersome to implement using the Legacy interface, but should be simple (and much more efficient of API calls for multiple-SAN certificates) when the driver has the whole list at once. Which is the sort of thing that motivated the new driver interface, of course.

[mmaney]

@DarkSuniuM @kylejohnson Any PowerDNS users working on this problem?