Allow multiple note types (footnotes, endnotes, etc.)

[adunning]

As discussed on the mailing list, it is possible in Word, Open/LibreOffice, LaTeX, and elsewhere to have both footnotes and endnotes in a single document, and publishers (examples from Oxford, Peeters) request that texts be submitted this way for various purposes (e.g. citations plus a glossary; separate textual and source notes in a scholarly edition). The final version might be rendered as footnotes plus endnotes (e.g. a commentary at the end of a text); multiple sets of footnotes or endnotes (often using some sort of differentiated marker system, e.g. letters in addition to numbers or a key to line numbers); or footnotes plus sidenotes.

There have also been requests in the past for sidenotes/margin notes. The consensus on the list seems to be that this could be useful, but isn't as necessary as simply having some ability to have two separate series of notes, especially given that the creation of this in other formats is less clear-cut outside LaTeX (marginpar), DocBook (sidebar; cf. the AsciiDoc implementation), and perhaps HTML (aside). It can be done only awkwardly in LibreOffice and Word, though I have heard of publishers using Word comments as text to be put in the margin. As another approach, Classical Text Editor allows as many as fifty separate series of notes (plus text to be placed in the inner and outer margins), each of which has separately-controlled rendering. This is surely too extreme for Pandoc (cf. the eledmac package in LaTeX), but it's an interesting approach.

It was suggested by @jkr that it could be structured like this:

data NoteType = Footnote | Endnote | Marginnote 

data Inline = ... 
            | Note NoteType [Block] 
            ... 

This is a summary of the list's proposals for the Markdown syntax, most of which centre around changing the character used for the footnote:

Written as footnotes

• [^1]: footnote and [#1]: endnote and [>1]: sidenote
• [^1]: footnote and [^^1]: endnote and [^^^1]: sidenote
• [^1]: footnote and [^a]: endnote (i.e. if different identifiers are being consistently used)

Written as inline notes

• ^[footnote] and ^^[endnote]
• ^[^ footnote] (the extra caret being optional) and ^[# endnote] and ^[> sidenote] and ^[| inline note]

[bpj]

• I don't think the [^1] vs. [^a] version will work as the current syntax for footnotes is actually [^arbitrary_identifier] so different 'magic characters' instead of/in addition to ^ is the way to go.
• One possible way of marking endnotes would be [~id] or [^~id] since Pandoc already uses ~sub~ vs. ^sup^.
• NB that it's not trivial to use very many \marginpars (as e.g. with margin glosses) in LaTeX since there is a finite but somewhat adjustable amount of memory allocated to them. In practice I have to use one marginpar with linebreaks for each paragraph.

[DanStaal]

A thought on [^1] vs [^a]: One thing it would allow would be to control whether footnotes, endnotes, or both were generated via markdown extensions. If the normal [^ is considered the base, then you can have a +endnotes option that turns [^a] into endnotes - without changing the file. Which means you get complete backwards compatibility.

[bpj]

The problem with [^a] vs. [^1] is that it breaks backwards
compatibility in a bad way. The possibility to use arbitrary note
identifiers is a valuable feature: while working on a document you can
use a descriptive id to easier find note references and definitions and
even more importantly if you already have reformatted your source with
pandoc and want to insert some more notes you don't need to check for
the highest already used id number or assign some arbitrary astronomic
number which anyway will be out of sequence, you just assign some
arbitrary descriptive id and have it turned into an in-sequence number.

Also how is [^a] vs. [^1] to interact with --id-prefix, which is really indispensable?

That's why I think the best way to indicate endnotes and sidenotes is to
add some punctuation after the caret. Since footnote ids can currently
contain any characters other than spaces, tabs and newlines (does that
mean they can contain carriagereturns? ;-) 'compatibility mode' would
just treat that extra punctuation as part of the id.

I don't like [^^endnote] which was my first thought anymore, but
[^#endnote] or [^~endnote] and the obvious [^>sidenote] would work
and be backwards compatible. It would also be more economic with
punctuation since [#foo] [~bar] [>baz] can be used for something else
later (cf. [@cite] -- I would particularly like [#anchor]! :-).

Another matter is that automatic numbering of endnotes should probably use the a..z,aa,ab..zy,zz series.

OTOH I think that the best change in the long run would be to store the
user-assigned note id in the AST. One could then use filters to convert
notes into endnotes, sidenotes or foonotes according to ones need on the
basis of the presence of some (sequences of) characters in the id. This
is also a change which @jgm has already expressed that he might
consider!

[cben]

Strawman idea: what if the location of the note body in the source affected its position in the output?
Something like:

# Section

Text [^marginnote] text [^footnote].

[^marginnote]: written following the reference more or less immediately.

More text [^endnote].
(How would I distinguish margin vs foot if this paragraph wasn't here?)

----

[^footnote]: collected at end of section.  This is ambiguos, requiring them to be below a `----` might be a good solution.

# New Section
...

# Endnotes

[^endnote]: collected at end of document?  A more robust criterion is being in another section -- people might want stuff after endnotes, e.g. a bibliography...

There are obviously holes in this proposal, and compatibility problems - until now all of these were footnotes of equal standing. But it's such a natural direction...

[jkr]

Seems fragile -- I know I would always be getting it wrong. Plus it would break concatting docs, which is pretty useful for multi-file documents.

My bigges interest is in having it in the AST -- I imagine you could filter to get there if you needed it, without disturbing most folks' use of just one note type. e.g.

Blah[^1]. Bleh[^2].


[^1]: default sort of note. Let's say footnote.
[^2]: <div class="endnote">I'm at the end.</div>

It's a pain to write. But the point is, it can be in the AST and useful for the small percentage who need it without needing to decide on markdown syntax for it. This was what happened with the generic divs and spans. The above also has the advantage of degrading gracefully.

That being said, if I had to vote for some sort of ascii markup, I would probably go for some replacement for the caret:

[^footnote]
[>marginnote]
[~endnote]

or the like.

[jkr]

Or, yeah, the backwards-compatible version in @bpj 's version above.

[DanStaal]

At least in the near term, at this point I'll agree the best is to get all of this info into the AST, so a filter can handle it. A fairly generic filter would then be a good thing for a 'pandoc extras' package.

There is the possibility of leaving the options in the filters - not everything has to be built into pandoc, if there is a good set of readily-available filters for 'common uncommon' things. It would keep pandoc simpler (trying to avoid the 'kitchen sink effect'), while still allowing the more powerful uses.

[jkr]

The only downside I'd see to leaving it in the filters is that you couldn't simulate the different-carets, since the AST doesn't keep the footnote marker. But other than that, I'd tend to agree. The only user-facing thing that would need to come with a change to the AST would be some sort of --default-note-type=... option.

I also like the idea of a pandoc-extras/pandoc-contrib filters package. A place to set up niche features, or try out some features on their way to inclusion.

This all makes me realize that I'd really like to see a filter option in the YAML block. In some docs I have a hefty amount of filters, but don't really want to set up a Makefile, for whatever reason. It would be nice to have the doc keep track of the filters it wants.

[bpj]

You could place a marker character as the first char of the footnote text, optionally followed by an all-alphanumeric id and then a space. Perhaps you could put it in braces so you won't need to backslash escape the marker. Not as elegant but the filter could see it, inspect it, and remove it from the note's content list before it possibly munges the note element. I've been thinking all day about whipping up a proof of concept, but I'm going away on Saturday and won't likely have time before then. Of course the real solution would be for all elements to take attributes -- namely as braces as the very first thing in the element, at the top of block elements, as the first thing after the delimiter in inline elements.

[adunning]

Interestingly, it turns out that someone has already made their own fork of Pandoc to accomplish this: https://github.com/susanemcg/pandoc-tufteLaTeX2GitBook.

[dfaligertwood]

I've also written some filters for this, in dfaligertwood/utility-scripts. They're not particularly well described, however (e.g. pandoc-musicnotes is actually second level textual annotations using edNotes.sty in LaTeX..)

I've used the first character of the footnote text to identify the type. This isn't really a brilliant method though, and my implementation is quite brittle. IMO the ideal method would be have either the footnote id or additional attributes available in the AST for filters.

[the-solipsist]

I think the best option would be to go ahead with the YAML method suggested by @jkr: #1425 (comment)

If this is within the YAML metadata, then this would allow for docx/odt -> markdown -> docx/odt conversion to work, otherwise, it would only be a one-way process.