Errors in documentation of standard codec error handlers

[RalfM]

BPO	20413
Nosy	@ncoghlan, @ezio-melotti, @vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-01-27.20:41:06.038>
labels = ['type-feature', 'docs']
title = 'Errors in documentation of standard codec error handlers'
updated_at = <Date 2015-02-07.00:01:07.843>
user = 'https://bugs.python.org/RalfM'

bugs.python.org fields:

activity = <Date 2015-02-07.00:01:07.843>
actor = 'ncoghlan'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2014-01-27.20:41:06.038>
creator = 'RalfM'
dependencies = []
files = []
hgrepos = []
issue_num = 20413
keywords = []
message_count = 5.0
messages = ['209477', '209502', '235496', '235500', '235506']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'ezio.melotti', 'mrabarnett', 'docs@python', 'martin.panter', 'RalfM']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20413'
versions = ['Python 3.3', 'Python 3.4', 'Python 3.5']

Linked PRs

• gh-64612: Update error handlers list under open() #137304

[RalfM]

The standard library documentation lists the standard codec error handlers in three places:

(a) 2. Build-in Functions, section open()
(b) 7.2 codecs - Codec registry and base classes
(c) 7.2.1 Codec Base Classes

As far as I can judge these lists, (c) looks ok, but (a) and (b) contain two errors:

1. 'surrogatepass' is not mentioned.
2. 'surrogateescape' is described as:
   'on decoding, replace with code points in the Unicode Private
   Use Area ranging from U+DC80 to U+DCFF. These private code points
   will ...'
   This is incorrect in so far as U+DC80 to U+DCFF are not private
   code points, but (low-)surrogate code points. This is correctly
   explained in (c) and in PEP-383 (and, of course, in the Unicode
   standard, chapter 16).

I suggest to correct (a) and (b) by

• adding 'surrogatepass' with the description given in (c),
• changing the description of 'surrogateescape' to something like:
  'on decoding, replace with surrogate code points ranging from
  U+DC80 to U+DCFF. These surrogate code points will ...'.

These errors are present in the documentation (more precisely, the .chm files) of at least

• Python 3.3.3
• Python 3.3.4rc1
• Python 3.4.0b3.

[ncoghlan]

I plan to take a look at the codec docs in general in the next week or so, I'll tackle this as well.

[mrabarnett]

The docs for Python 3.5.0a0 still say "Unicode Private Use Area".

[vadmium]

I changed “code point in the Unicode Private Use Area” to “individual surrogate code” in the “codecs” module documentation for bpo-19548. So perhaps (a) still needs addressing, but (b) and (c) are hopefully already fixed.

[ncoghlan]

Ah, February 2014, many of my plans went in rather different directions than expected that month, and this was one of them :)

As Martin noted, he already fixed (b) and (c), but we missed that the list of error handlers was also duplicated in the builtin open() docs.

That duplication is likely worthwhile from a docs usability perspective, but we should:

1. Bring it in line with Martin's recent fixes to the codecs module docs
2. Add a comment in the error handler docs noting that the open() docs may need to be updated to reflect changes to error handler semantics