Skip to content

Add support for Mailtrap #406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add support for Mailtrap #406

wants to merge 2 commits into from

Conversation

cahna
Copy link

@cahna cahna commented Nov 10, 2024

Add support for Mailtrap

@cahna cahna force-pushed the mailtrap branch 5 times, most recently from 8c44826 to 1ef637c Compare November 10, 2024 23:55
@medmunds
Copy link
Contributor

Thanks for this! It looks pretty good at first glance; I'll try to take a closer look later this week.

Mailtrap does not support: inbound emails and reply-to.

Missing inbound support is not unusual; it's fine to just ignore it.

Reply-to is a little surprising. Some ESPs require handling this as an extra header. (I haven't had a chance to look at Mailtrap's docs.)

I am also having trouble getting local builds and tests to work

What's going wrong? (In the contributing docs, I notice the "test a representative combination of Python and Django versions" command is outdated—the current version should be tox -e lint,django51-py312-all,django40-py38-all,docs. But other than that I'd expect it to work. I'm usually on macOS, but the GitHub tests all run on Ubuntu.)

@medmunds medmunds marked this pull request as draft November 12, 2024 02:34
@cahna
Copy link
Author

cahna commented Nov 17, 2024

@medmunds

Reply-to is a little surprising. Some ESPs require handling this as an extra header.

Aha! I'm no email expert, so TIL! I will add the reply-to-via-headers support momentarily.

@cahna
Copy link
Author

cahna commented Nov 17, 2024

@medmunds

What's going wrong? (In the contributing docs, I notice the "test a representative combination of Python and Django versions" command is outdated—the current version should be tox -e lint,django51-py312-all,django40-py38-all,docs. But other than that I'd expect it to work. I'm usually on macOS, but the GitHub tests all run on Ubuntu.)

Switching to a dev container, using pyenv to install 3.12 and 3.8, and using the updated tox command got further. I got a complaint that python 3.8 was missing, but creating a symlink /usr/bin/python3.8 to the pyenv installed version fixed that. I think I'm good to go for writing tests. Thanks.

@cahna cahna force-pushed the mailtrap branch 2 times, most recently from 18307bc to ab47bbc Compare November 17, 2024 22:28
Copy link
Contributor

@medmunds medmunds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cahna this is looking really solid! I have a bunch of suggestions, some responses to your questions, and some questions of my own.

I haven't been through the webhook you just added yet; will take a look at that soon.

[GitHub sometimes hides review comments: please search the page for "Hidden Conversations" to be sure you see them all.]

Copy link
Contributor

@medmunds medmunds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, the webhooks look really good, too. Handful of minor comments.

Thanks again for your work on this.

@cahna
Copy link
Author

cahna commented Nov 24, 2024

Thanks for your feedback @medmunds! Taking a look now.

FYI: I only have Sundays available to work on this for the foreseeable future.

@cahna
Copy link
Author

cahna commented Nov 24, 2024

Searching for "hidden" within the page showed nothing, so I think I saw everything. LMK if I missed something.

@cahna
Copy link
Author

cahna commented Nov 24, 2024

I will start working on integration tests locally with my own free account. @medmunds, would you be able to setup a free mailtrap account, put the API key and test inbox ID into the repo's github secrets, and expose them to the build as environment variables?

FYI: the free mailtrap tier allows 100 test emails/month.

parsed_response = self.deserialize_json_response(response, payload, message)

# TODO: how to handle fail_silently?
if not self.fail_silently and (
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While trying to duplicate test coverage from the other backend tests, I found that the fail_silently test(s) don't work, and this doesn't seem to be the correct way to do this. How should I handle that? Should I?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fail_silently should be handled for you in AnymailBaseBackend.send_messages(). The important part is that any errors raised by this function need to be one of the Anymail errors, not generic Python errors like KeyError.

It's helpful to distinguish a few different cases:

  • If the API uses HTTP status codes to indicate an error, that's already been reported in AnymailRequestsBackend.raise_for_status() (so parse_recipients_status() won't be called). It looks to me like that's the case for all Mailtrap error responses.
  • (If the API indicates errors by returning HTTP 2xx but with an "errors" field in the response, that needs to be reported as an AnymailRequestsAPIError with the message extracted from the response. This doesn't seem to apply to Mailtrap.)
  • If the API indicated success, but the response isn't parseable (e.g., missing keys), that should be reported as an AnymailRequestsAPIError with a message describing the invalid response format. I'll add a separate review comment with a suggestion for that.

@cahna
Copy link
Author

cahna commented Nov 30, 2024

@medmunds I got started on backend unit tests, but I have left several @unittest.skip("TODO: ...") on tests that either might not be relevant for Mailtrap, or that have outstanding questions on how to adjust the backend code to make the test pass.

@cahna cahna requested a review from medmunds November 30, 2024 20:18
@medmunds
Copy link
Contributor

@medmunds, would you be able to setup a free mailtrap account, put the API key and test inbox ID into the repo's github secrets, and expose them to the build as environment variables?

Done: variables ANYMAIL_TEST_MAILTRAP_DOMAIN and ANYMAIL_TEST_MAILTRAP_TEST_INBOX_ID and secret ANYMAIL_TEST_MAILTRAP_API_TOKEN are installed. (Secrets aren't available to PR builds, so the integration tests won't actually run until it's merged into main.)

Copy link
Contributor

@medmunds medmunds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still looking great! And nice start on the docs.

Handful of feedback on the main implementation.

I glanced through the backend tests and tried to respond to your questions. This is the part of adding an ESP where we typically discover all of the little details they haven't documented, so some tests may need adjustment based on that. (The "test_all_options" integration test can be helpful for discovering those.)

(Apologies for not getting feedback on your earlier round sooner—I'm starting to get caught up in the end of the year rush.)

parsed_response = self.deserialize_json_response(response, payload, message)

# TODO: how to handle fail_silently?
if not self.fail_silently and (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fail_silently should be handled for you in AnymailBaseBackend.send_messages(). The important part is that any errors raised by this function need to be one of the Anymail errors, not generic Python errors like KeyError.

It's helpful to distinguish a few different cases:

  • If the API uses HTTP status codes to indicate an error, that's already been reported in AnymailRequestsBackend.raise_for_status() (so parse_recipients_status() won't be called). It looks to me like that's the case for all Mailtrap error responses.
  • (If the API indicates errors by returning HTTP 2xx but with an "errors" field in the response, that needs to be reported as an AnymailRequestsAPIError with the message extracted from the response. This doesn't seem to apply to Mailtrap.)
  • If the API indicated success, but the response isn't parseable (e.g., missing keys), that should be reported as an AnymailRequestsAPIError with a message describing the invalid response format. I'll add a separate review comment with a suggestion for that.

Comment on lines 246 to 265
# TODO: how to handle fail_silently?
if not self.fail_silently and (
not parsed_response.get("success")
or ("errors" in parsed_response and parsed_response["errors"])
or ("message_ids" not in parsed_response)
):
raise AnymailRequestsAPIError(
email_message=message, payload=payload, response=response, backend=self
)
else:
# message-ids will be in this order
recipient_status_order = [
*payload.recipients_to,
*payload.recipients_cc,
*payload.recipients_bcc,
]
recipient_status = {
email: AnymailRecipientStatus(
message_id=message_id,
status="sent",
)
for email, message_id in zip(
recipient_status_order, parsed_response["message_ids"]
)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an approach for reporting invalid responses. It's also probably worth checking the number of message_ids, since zip() won't raise an error for that.

Suggested change
# TODO: how to handle fail_silently?
if not self.fail_silently and (
not parsed_response.get("success")
or ("errors" in parsed_response and parsed_response["errors"])
or ("message_ids" not in parsed_response)
):
raise AnymailRequestsAPIError(
email_message=message, payload=payload, response=response, backend=self
)
else:
# message-ids will be in this order
recipient_status_order = [
*payload.recipients_to,
*payload.recipients_cc,
*payload.recipients_bcc,
]
recipient_status = {
email: AnymailRecipientStatus(
message_id=message_id,
status="sent",
)
for email, message_id in zip(
recipient_status_order, parsed_response["message_ids"]
)
}
try:
message_ids = parsed_response["message_ids"]
except KeyError as err:
raise AnymailRequestsAPIError(
"Invalid response format: missing message_ids",
email_message=message, payload=payload, response=response, backend=self
) from err
# message-ids will be in this order
recipient_status_order = [
*payload.recipients_to,
*payload.recipients_cc,
*payload.recipients_bcc,
]
if len(recipient_status_order) != len(message_ids):
raise AnymailRequestsAPIError(
"Invalid response format: wrong number of message_ids",
email_message=message, payload=payload, response=response, backend=self
)
recipient_status = {
email: AnymailRecipientStatus(
message_id=message_id,
status="sent",
)
for email, message_id in zip(recipient_status_order, message_ids)
}

API error responses should already be covered by raise_for_status() in the superclass. (But if you wanted to add a check like if parsed_response.get("errors") or not parsed_response.get("success"): raise AnymailRequestsAPIError("Unexpected errors in {response.status_code} response", ...), that wouldn't hurt anything.)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my integration testing, it seems that there is always exactly one message id returned in message_ids. Can you find or think of any ways this would return more than one message_id?

I have updated the code to expect only one message id in the response and associated it with all addresses.

@unittest.skip("TODO: is this test correct/necessary?")
def test_send_with_recipients_refused(self):
"""Test sending an email with all recipients refused"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably unnecessary, since Mailtrap doesn't seem to provide per-recipient send status.

Mailtrap's error documentation isn't all that descriptive, so we might need to test this. If you try to send a message with two recipients, both on your suppression list (or previous bounces), what happens? (Send API success and later webhook reject? Or send API error?)

Similarly, if you send with one valid recipient and one suppressed recipient, what happens?

(I can do some investigation here if that would be helpful.)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have removed this test, but I have not investigated/tested the cases you mentioned.

mail.send_mail("Subject", "Message", "[email protected]", ["[email protected]"])
errmsg = str(cm.exception)
self.assertRegex(errmsg, r"\bMAILTRAP_API_TOKEN\b")
self.assertRegex(errmsg, r"\bANYMAIL_MAILTRAP_API_TOKEN\b")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably also need to test various combinations of settings, both valid and error cases (api_url, testing_enabled, test_inbox_id), to make sure the right api is called or a helpful error is raised.

@adamwolf
Copy link

adamwolf commented May 6, 2025

Hiya folks!

Is there I could help with to help get this ready?

@medmunds
Copy link
Contributor

@cahna did you still want to work on this? If not, I'll try to finish it up. (Though realistically, probably not until sometime next month.)

@adamwolf thanks for the offer. There are a couple of things that would be helpful:

  • We need to know what Mailtrap's send API response looks like when you try to send to blocked addresses or mixed blocked/not-blocked. (See comment above.)
  • If cahna isn't planning to move forward with this PR, you're welcome to adopt it and continue in a new PR. It would probably be helpful to squash the current commits (maintaining co-author credit for cahna) and continue from there with the unresolved review feedback above.

@cahna
Copy link
Author

cahna commented May 11, 2025

My apologies for ghosting. Work and life has been very busy. Good news is that this has been running in a production environment for 3+ months without any issues (nearing 2 million emails). I want to finish up the integration tests and outstanding comments, but i can't commit to a deadline at this point.

@cahna
Copy link
Author

cahna commented Aug 13, 2025

I have a question. For this integration test case:

    def test_all_options(self):
        message = AnymailMessage(
            subject="Anymail Mailtrap all-options integration test",
            body="This is the text body",
            from_email=formataddr(("Test From, with comma", self.from_email)),
            to=[
                "[email protected]",
                'Recipient 2 <[email protected]>',
            ],
            cc=["[email protected]", "Copy 2 <[email protected]>"],
            bcc=["[email protected]", "Blind Copy 2 <[email protected]>"],
            reply_to=[
                '"Reply, with comma" <[email protected]>',
                "[email protected]",
            ],
            headers={"X-Anymail-Test": "value", "X-Anymail-Count": "3"},
            metadata={"meta1": "simple string", "meta2": 2},
            # Mailtrap supports only a single tag/category
            tags=["tag 1"],
            track_clicks=True,
            track_opens=True,
        )
        message.attach("attachment1.txt", "Here is some\ntext for you", "text/plain")
        message.attach("attachment2.csv", "ID,Name\n1,Amy Lina", "text/csv")
        cid = message.attach_inline_image_file(sample_image_path())
        message.attach_alternative(
            "<p><b>HTML:</b> with <a href='http://example.com'>link</a>"
            f"and image: <img src='cid:{cid}'></div>",
            "text/html",
        )

        message.send()
        self.assertEqual(message.anymail_status.status, {"sent"})
        self.assertEqual(
            message.anymail_status.recipients["[email protected]"].status, "sent"
        )
        self.assertEqual(
            message.anymail_status.recipients["[email protected]"].status, "sent"
        )

Mailtrap's sandbox receives this raw email:

MIME-Version: 1.0
Date: Wed, 13 Aug 2025 19:55:17 +0000
Reply-To: "Reply, with comma" <[email protected]>, [email protected]
X-Anymail-Test: value
X-Anymail-Count: 3
Subject: Anymail Mailtrap all-options integration test
From: "Test From, with comma" <[email protected]>
To: [email protected], "Recipient 2" <[email protected]>
Cc: [email protected], "Copy 2" <[email protected]>
Content-Type: multipart/mixed;
 boundary=92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9

--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9
Content-Type: multipart/related;
 boundary=270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5

--270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5
Content-Type: multipart/alternative;
 boundary=326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a

--326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

This is the text body
--326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<p><b>HTML:</b> with <a href=3D'http://example.com'>link</a>and image: <img=
 src=3D'cid:175511491762.9071.11355961105641567026.img@inline'></div>
--326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a--

--270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5
Content-Disposition: inline; filename="sample_image.png"
Content-ID: <175511491762.9071.11355961105641567026.img@inline>
Content-Transfer-Encoding: base64
Content-Type: image/png

iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz
AAALEgAACxIB0t1+/AAAABR0RVh0Q3JlYXRpb24gVGltZQAzLzEvMTNoZNRjAAAAHHRFWHRTb2Z0
d2FyZQBBZG9iZSBGaXJld29ya3MgQ1M1cbXjNgAAAZ1JREFUWIXtl7FKA0EQhr+TgIFgo5BXyBUp
fIGksLawUNAXWFFfwCJgBAtfIJFMLXgQn8BSwdpCiPcKAdOIoI2x2Dmyd7kYwXhp9odluX/uZv6d
nZu7DXowxiKZi0IAUHKCvxcsoAIEpST4IawVGb0Hb0BlpcigefACvAAvwAsoTTGGlwwzBAyivLUP
EZrOM10AhGOH2wWugVVlHoAdhJHrPC8DNR0JGsAAQ9mxNzBOMNjS4Qrq69U5EKmf12ywWVsQI4QI
IbCn3Gnmnk7uk1bokfooI7QRDlQIGCdzPwiYh0idtXNs2zq3UqwVEiDcu/R0DVjUnFpItuPSscfA
FXCGSfEAdZ2fVeQ68OjYWwi3ycVvMhABGwgfKXZScHeZ+4c6VzN8FbuYukvOykCs+z8PJ0xqIXYE
d4ALoKlVH2IIgUHWwd/6gNAFPjPcCPvKNTDcYAj1lXzKc7GIRrSZI6yJzcQ+dtV9bD+IkHThBj34
4j9/yYxupaQbXPJLNqsGFgeZ6qwpLP1b4AV4AV5AoKfjpR5OwR6VKwULCAC+AQV4W9Ps4uZQAAAA
AElFTkSuQmCC
--270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5--

--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9
Content-Disposition: attachment; filename="attachment1.txt"
Content-Transfer-Encoding: base64
Content-Type: text/plain

SGVyZSBpcyBzb21lCnRleHQgZm9yIHlvdQ==
--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9
Content-Disposition: attachment; filename="attachment2.csv"
Content-Transfer-Encoding: base64
Content-Type: text/csv

SUQsTmFtZQoxLEFteSBMaW5h
--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9--

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

@cahna
Copy link
Author

cahna commented Aug 13, 2025

For the integration tests of templates, would you please create a template in your mailtrap account?

Needed steps:

  1. Go to templates to add a new template with these details:
    image
  2. Select "Welcome email" and leave it unchanged:
    image
  3. Click "Finish":
    image
  4. Copy the template UUID:
    image
  5. Add the template UUID to github actions variables as ANYMAIL_TEST_MAILTRAP_TEMPLATE_UUID

@cahna
Copy link
Author

cahna commented Aug 13, 2025

It appears that bcc testing is not available on their free plan:

image

@cahna cahna force-pushed the mailtrap branch 3 times, most recently from 0a76d0a to bc1fd1d Compare August 13, 2025 20:36
@cahna cahna requested a review from medmunds August 13, 2025 20:36
@cahna
Copy link
Author

cahna commented Aug 13, 2025

My conscience continues to make me feel bad about not finishing this, so I have added minimal integration tests, added fixes for some of the PR comments, squashed the commits, and rebased on latest main branch.

@medmunds
Copy link
Contributor

@cahna thanks! I re-rebased to the latest main, so you should pull before any other changes.

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

That sounds correct: if there's only a single message id, it applies to all recipients. I'll try to do some live testing to confirm. (The main goal is have an ID that could be matched with a later status webhook call somehow.)

It appears that bcc testing is not available on their free plan:

That's an interesting pricing differentiator. (Maybe bcc was getting used by spammers? I think there's another ESP who limits bcc recipients to validated domains for that very reason.)

We can just omit bcc from the integration test.

I'd be inclined to label Mailtrap as "Full" support in the feature matrix: we're able to perform integration testing (on everything except bcc) and the 3500 messages/month on the free plan will way more than cover Anymail's test sending.

@medmunds
Copy link
Contributor

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

That sounds correct: if there's only a single message id, it applies to all recipients. I'll try to do some live testing to confirm.

Hmm, If I send to two "to" and two "cc" recipients on a production domain (https://send.api.mailtrap.io/api/send endpoint), I get this response, which seems to mean your zip code is correct:

{
  "success": true,
  "message_ids": [
    "ed1edba0-795c-11f0-0000-f14311a5578e",
    "ed1fc600-795c-11f0-0000-f14311a5578e",
    "ed20fe80-795c-11f0-0000-f14311a5578e",
    "ed21c1d0-795c-11f0-0000-f14311a5578e"
  ]
}

But on a sandbox domain (https://sandbox.api.mailtrap.io/api/send/SANDBOX_ID), I get a single message_id:

{
  "success": true,
  "message_ids": [
    "5040626968"
  ]
}

@cahna
Copy link
Author

cahna commented Aug 15, 2025

the 3500 messages/month on the free plan will way more than cover Anymail's test sending

@medmunds Just a heads-up that the "sandbox" is only limited to 100/month

@cahna
Copy link
Author

cahna commented Aug 15, 2025

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

That sounds correct: if there's only a single message id, it applies to all recipients. I'll try to do some live testing to confirm.

Hmm, If I send to two "to" and two "cc" recipients on a production domain (https://send.api.mailtrap.io/api/send endpoint), I get this response, which seems to mean your zip code is correct:

{
  "success": true,
  "message_ids": [
    "ed1edba0-795c-11f0-0000-f14311a5578e",
    "ed1fc600-795c-11f0-0000-f14311a5578e",
    "ed20fe80-795c-11f0-0000-f14311a5578e",
    "ed21c1d0-795c-11f0-0000-f14311a5578e"
  ]
}

But on a sandbox domain (https://sandbox.api.mailtrap.io/api/send/SANDBOX_ID), I get a single message_id:

{
  "success": true,
  "message_ids": [
    "5040626968"
  ]
}

Hmm... strange behavior. What do you think would be best to do here? If self.testing_enabled expect one ID and use it for all recipients, but if not, use the zip code?

I have only been using Anymail to send single-recipient emails, so this is not something I had encountered, yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants