The email package should defer to the codecs module for all aliases

[bitdancer]

BPO	8898
Nosy	@malemburg, @warsaw, @vstinner, @ezio-melotti, @merwok, @bitdancer, @mmaker
Files	issue8898.patch fail_tactis.txt issue8898_withtests.patch fail_mcbs.txt issue8898_skip.patch issue8898_normalize.patch issue8898_2.patch issue8898_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-06-04.18:53:49.968>
labels = ['type-feature', 'expert-email']
title = 'The email package should defer to the codecs module for\tall aliases'
updated_at = <Date 2019-07-29.12:01:12.918>
user = 'https://github.com/bitdancer'

bugs.python.org fields:

activity = <Date 2019-07-29.12:01:12.918>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2010-06-04.18:53:49.968>
creator = 'r.david.murray'
dependencies = []
files = ['22053', '22058', '22059', '22060', '22066', '22094', '22146', '22153']
hgrepos = []
issue_num = 8898
keywords = ['patch']
message_count = 39.0
messages = ['107087', '107093', '107098', '107100', '107102', '124713', '136443', '136488', '136507', '136511', '136514', '136515', '136518', '136519', '136520', '136521', '136533', '136539', '136550', '136551', '136553', '136614', '136636', '136764', '136984', '136989', '136994', '136996', '136998', '136999', '137007', '137048', '137049', '137051', '137056', '137060', '137072', '137082', '348647']
nosy_count = 8.0
nosy_names = ['lemburg', 'barry', 'vstinner', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'l0nwlf', 'maker']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue8898'
versions = ['Python 3.3']

Linked PRs

• gh-137153: Add latin_N aliases to encodings.aliases #136701

[bitdancer]

Currently the email module maintains a set of "charset" aliases that it maps to codec names before looking up the codec in the codecs module. Ideally it should instead be able to just look up any 'charset' name, and if it is a valid alias for a codec, the codec module would return the codec with the canonical name. It is possible (I haven't checked yet) that the email module needs a different canonical 'charset' name for certain codecs, but if so it can do that mapping after getting the canonical codec name from codecs.

To implement this we need to make two simple changes:

1. add any aliases the email module recognizes but the codecs module doesn't to the codecs module.

2. rewrite email.charset so that it does not have an ALIASES table (but may have a smaller 'canonical charset map' table instead).

[l0nwlf]

from email.charset.ALIASES most of them failed to be recognize by codecs module.

>>> for i in email.charset.ALIASES.keys():
...     try:
...         codecs.lookup(i)
...     except LookupError:
...         print("Not recognized by codecs : alias {} mapped to {}".format(i, email.charset.ALIASES[i]))
...     
... 
Not recognized by codecs : alias latin-8 mapped to iso-8859-14
Not recognized by codecs : alias latin-9 mapped to iso-8859-15
Not recognized by codecs : alias latin-2 mapped to iso-8859-2
Not recognized by codecs : alias latin-3 mapped to iso-8859-3
<codecs.CodecInfo object for encoding iso8859-1 at 0x10160af58>
Not recognized by codecs : alias latin-6 mapped to iso-8859-10
Not recognized by codecs : alias latin-7 mapped to iso-8859-13
Not recognized by codecs : alias latin-4 mapped to iso-8859-4
Not recognized by codecs : alias latin-5 mapped to iso-8859-9
<codecs.CodecInfo object for encoding euc_jp at 0x1016260b8>
Not recognized by codecs : alias latin-10 mapped to iso-8859-16
<codecs.CodecInfo object for encoding ascii at 0x101626120>
Not recognized by codecs : alias latin_10 mapped to iso-8859-16
<codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0>
Not recognized by codecs : alias latin_2 mapped to iso-8859-2
Not recognized by codecs : alias latin_3 mapped to iso-8859-3
Not recognized by codecs : alias latin_4 mapped to iso-8859-4
Not recognized by codecs : alias latin_5 mapped to iso-8859-9
Not recognized by codecs : alias latin_6 mapped to iso-8859-10
Not recognized by codecs : alias latin_7 mapped to iso-8859-13
Not recognized by codecs : alias latin_8 mapped to iso-8859-14
Not recognized by codecs : alias latin_9 mapped to iso-8859-15
<codecs.CodecInfo object for encoding cp949 at 0x101626390>
<codecs.CodecInfo object for encoding euc_kr at 0x101626530>

So basically apart from latin-1 all the latin* failed to be recognized by codecs.

[malemburg]

Shashwat Anand wrote:
> 
> Shashwat Anand <anand.shashwat@gmail.com> added the comment:
> 
> from email.charset.ALIASES most of them failed to be recognize by codecs module.
> 
> 
>>>> for i in email.charset.ALIASES.keys():
> ...     try:
> ...         codecs.lookup(i)
> ...     except LookupError:
> ...         print("Not recognized by codecs : alias {} mapped to {}".format(i, email.charset.ALIASES[i]))
> ...     
> ... 
> Not recognized by codecs : alias latin-8 mapped to iso-8859-14
> Not recognized by codecs : alias latin-9 mapped to iso-8859-15
> Not recognized by codecs : alias latin-2 mapped to iso-8859-2
> Not recognized by codecs : alias latin-3 mapped to iso-8859-3
> <codecs.CodecInfo object for encoding iso8859-1 at 0x10160af58>
> Not recognized by codecs : alias latin-6 mapped to iso-8859-10
> Not recognized by codecs : alias latin-7 mapped to iso-8859-13
> Not recognized by codecs : alias latin-4 mapped to iso-8859-4
> Not recognized by codecs : alias latin-5 mapped to iso-8859-9
> <codecs.CodecInfo object for encoding euc_jp at 0x1016260b8>
> Not recognized by codecs : alias latin-10 mapped to iso-8859-16
> <codecs.CodecInfo object for encoding ascii at 0x101626120>
> Not recognized by codecs : alias latin_10 mapped to iso-8859-16
> <codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0>
> Not recognized by codecs : alias latin_2 mapped to iso-8859-2
> Not recognized by codecs : alias latin_3 mapped to iso-8859-3
> Not recognized by codecs : alias latin_4 mapped to iso-8859-4
> Not recognized by codecs : alias latin_5 mapped to iso-8859-9
> Not recognized by codecs : alias latin_6 mapped to iso-8859-10
> Not recognized by codecs : alias latin_7 mapped to iso-8859-13
> Not recognized by codecs : alias latin_8 mapped to iso-8859-14
> Not recognized by codecs : alias latin_9 mapped to iso-8859-15
> <codecs.CodecInfo object for encoding cp949 at 0x101626390>
> <codecs.CodecInfo object for encoding euc_kr at 0x101626530>
> 
> 
> So basically apart from latin-1 all the latin* failed to be recognized by codecs.

We need to add aliases for those codecs. The current aliases
list only supports the format "latinN" for N in 1-10.

[l0nwlf]

We need to add aliases for those codecs. The current aliases
list only supports the format "latinN" for N in 1-10.

latinN means latin1 to latin10 ?
But latin_1 is a recognized alias.

>>> codecs.lookup('latin_1')
<codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0>

[malemburg]

Shashwat Anand wrote:

Shashwat Anand <anand.shashwat@gmail.com> added the comment:

> We need to add aliases for those codecs. The current aliases
> list only supports the format "latinN" for N in 1-10.

latinN means latin1 to latin10 ?

Yes. We should add aliases for the format "latin_N" as well.

But latin_1 is a recognized alias.

>>> codecs.lookup('latin_1')
<codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0>

Yes, since that's the native name of the dedicated Python codec
for ISO-8859-1.

[bitdancer]

Too late for 3.2, will implement for 3.3.

[mmaker]

The attached patch adds aliases for latin_N in encodings.aliases, and fixes email.charset behaviour according to codecs.lookup, as requested.
Tested on (Arch) Linux.

Am I supposed to add any unittest? I'm wavering about where they should be placed (in encodings or email?).

[ezio-melotti]

The patch looks ok to me.
AFAIU the lookup will take care to normalize the name and return latin_N. This also implies that other names (like 'latin-N', 'LaTiN~~N' and so on) will be normalized to latin_N and then accepted.

Regarding the tests, I don't see tests for the aliases anywhere, so something like:
for alias, codec_name in encodings.aliases.items():
self.assertEqual(codecs.lookup(alias).name, codec_name)
could be added somewhere to check that all the aliases in the dict map to the correct codec.

[mmaker]

Well, actually encodings.aliases links to the encoding _module name_, as
described in the doc:
""" Encoding Aliases Support
This module is used by the encodings package search function to
map encodings names to module names.
"""
So I've adjusted your snippet according to this, as you can see in the
attachment.

I've also slightly changed the imports as PEP-8 says:
"""
Yes: import os
import sys

No: import sys, os
"""

Anyway, running the test failed for two encodings, there are two bugs there,
indeed.

• mcbs has something broken in its imports;
• tactis module is not present.

Since they are really easy to fix, I haven't yet reported to the bugtraker.
Let me know what should I do.
Post on bugs.python.org bug and patch? Any new test specifically for the
email module?