Pandoc WASM

[tarleb]

For those interested: @TerrorJack successfully compiled pandoc to WASM via ghc-wasm-meta.

• Livedemo: https://tweag.github.io/pandoc-wasm/
• Repo: https://github.com/tweag/pandoc-wasm

Fantastic work!

[fiapps]

It's too bad Lua filters don't work yet. This makes it less useful.

I experimented with it briefly using the iOS app a-Shell. I copied the URL of pandoc.wasm from the README and downloaded it with curl. When I run it, it hangs, but if I press CTRL-C, it produces the expected output. The hang isn't due to an attempt to read from stdin, because CTRL-D has no effect. Probably, it needs to be build with a-Shell's custom WASI-SDK, and perhaps some of TerrorJack's customization needs to be undone. Since this is not wasmtime but another platform, it may be that Lua filters can be made to work. However, I don't expect to dedicate time to this in the near future. Since I have no experience with Haskell, I imagine I'd have a hard time making a custom build of pandoc.

[jgm]

@TerrorJack this is amazing!

I'm wondering whether we should explore modifying the main pandoc repo with a wasm build configuration, so that a separate fork is not needed -- perhaps with CPP conditionals. Do you think this would be worthwhile?

[jgm]

This is encouraging. Because our build process already allows you to build pandoc without Lua support, we could provide both a portable wasm without Lua support and a less portable one with Lua support.

@tarleb would be able to say whether we ever use os_tmpname.

[tarleb]

I believe that os.tmpname is almost never used in third-party scripts, and definitely not in our scripts. The most common use-cases require a temporary directory, which can be generated via pandoc.system.with_temporary_directory, to that's what most people use. But even then, most filters that use tempfiles of any kind would require additional software and wouldn't work on wasm anyway, so it seems perfectly fine to break them and just ignore tempfiles.

[jgm]

lua makes use of some posix functions that are not present on wasi, the most troublesome one is tmpnam. This can be fixed with either assuming /tmp is present (and bringing in non portable assumption about the virtual fs is provided by the wasm runtime), or letting os_tmpname always bail out with error. I prefer the latter, though I don't know how large it might impact downstream users like pandoc filters.

Is it possible to have conditional behavior? That is, check for /tmp in the vfs, and if it is not present, bail with error? But I agree with @tarleb that this shouldn't be a big issue even if we error (provided tmpnam isn't required for the Lua interpreter itself).

lua makes use of setjmp/longjmp. This is already supported by wasi-sdk by lowering to legacy eh opcodes, supported by chrome/firefox/safari, but it's not supported by wasmtime, so it hurts the portability of wasm module a bit.

Is it possible that support will be added to wasmtime?

isocline does not work for wasm32-wasi and needs to be patched, you can't expect anything terminal related functionality unless you're rolling custom tty functionality outside of wasi standard using something like xterm.js

We don't need the Lua repl in the wasm version, I think, and we could put support for it behind a cabal flag.

[tarleb]

I believe @cscheid has been doing some experiments with Lua in wasm, too. @cscheid, do you have some additional insights on these issues?

[jgm]

In my pandoc-wasm branch I have now added a repl flag on pandoc-cli and pandoc-lua-engine; this is disabled for wasm builds, and that solves the problem with isocline, at least. The next compilation error comes from hslua: src/Lua/Constants.hsc:107 directive const_str cannot be handled in cross-compilation mode. [EDIT: aha, found the cross-compile flag on lua, which fixes this!]

[jgm]

Looking at this again. I'd love to make some changes that would make pandoc easy to compile to wasm.

For reference, here are the changes @TerrorJack had to make to the pandoc code: main...haskell-wasm:pandoc:wasm

@TerrorJack has the situation changed much since 2024? Would we still need a long cabal.project with special flags for these various dependencies, and in many cases forks? Or have we gotten to the point where most of the dependencies don't need special treatment?

[TerrorJack]

i'll revisit it after holiday season. last time i was too deep into rabbit hole of lua support that i got burnt out halfway, maybe we could settle down on upstreaming without lua first. re dependencies, i think we can live with less wasm-specific dependencies these days, we'll see once i do a second try.

[jgm]

I have put some work into this. See the wasm branch in this repository for a very slimmed down version of @TerrorJack's commit that adds support for compiling to wasm. I removed things in cabal.project that no longer seem necessary, and the new http build flag in pandoc.cabal allows us to avoid the more intrusive commits in the code. (For now, Lua support is disabled.) Unfortunately we still do seem to need forked versions of several dependencies, and these are probably now a year or more behind the current versions.

@TerrorJack, currently the playground only works for text formats (you can't produce or read a docx, for example). If you try to produce a binary format, you get the output "Type error" -- I'm not sure where this is coming from exactly, and I don't know if it's a limitation in pandoc.wasm or just in the JS interface. But it would be good to have an interface capable of handling binary formats. One thought I had is that we could have the pandoc.wasm interface read and write MIME.

[jgm]

I have got a version of the playground now that (i) works with JSON-encoded options rather than command-line syntax, (ii) allows you to upload files, and (iii) allows you to specify an "output-file" (which is then added to the list of uploaded files and can be downloaded from there). Binary formats work for input and output files. This unlocks most of pandoc's functionality, though Lua filters are still not supported. I'll make a demo available when I have a chance.

[jgm]

Here's a demo of my progress building on @TerrorJack 's work.
https://pandoc.org/wasm-demo/
The demo has a clunky interface; you just specify a JSON version of a defaults file, and upload any input files and auxiliary files you need. But it shows that the core functions work.

[simonmichael]

love it!

[jgm]

Here's a demo of a user-friendly wrapper around pandoc.wasm, produced in 10 minutes by Claude: https://pandoc.org/wasm-demo/convert.html

[jgm]

@TerrorJack if you have time to have a quick look
https://github.com/jgm/pandoc/blob/wasm/cabal.project#L35-L119
I would love to get this compiling with Lua support, even if we have to make various compromises. Currently the build is failing with linker errors; I suspect something very basic is missing, but I don't understand all the pieces.

We're using this flake to get access to ghc-wasm-meta:
https://github.com/jgm/pandoc/blob/wasm/flake.nix#L1-L80

To trigger building with Lua it's sufficient to change -lua to +lua under pandoc-cli, and then make pandoc.wasm.

[kamelkace]

It looks like an error handler somewhere in the bytecode is causing the optimizer's flatten stage to fail. This is what I'm seeing in rr (source, albeit, a few lines away since curr is optimized out by line 231):

(rr) call curr->dump()
(try_table (catch $tag$0 $block2)
 (unreachable)
)

From poking around in the debug output of wasm-opt/binaryen, I don't think there's a preprocessing stage that is creating this in the bytecode, so it's probably either (1) a bug in binaryen's handling of exception handlers, or (2) a bug in the WASM bytecode generation of something else in the stack (or both, I suppose). Diagnosing that requires more context than I have, but adding --strip-eh (per the docs, thrown exceptions will now crash) seems to get the optimizer working. Not sure if that'll break something else, however.

# example
wasm-opt -d --low-memory-unused --converge --gufa --strip-eh --flatten --rereloop -Oz $(wasm32-wasi-cabal list-bin pandoc-cli | tail -1) -o pandoc.wasm

[jgm]

To keep things simple for now, I think I'll just strip it down to -Oz. That gives me a 56M binary instead of 54M with no error.

[jgm]

Added a Lua filters example: https://pandoc.org/wasm-demo/
It is working!

[simonmichael]

Congrats! The demo is really featureful and impressive and the examples are very helpful. A lovely addition would be showing the equivalent command line.

[tarleb]

Amazing! 🚀

[simonmichael]

Is https://pandoc.org/wasm-demo broken today ? It spins forever at "Loading pandoc.wasm..." for me.

[simonmichael]

Just me I think; switching to my normal wifi fixed it.

[jgm]

It was indeed broken for a short time.

[tarleb]

I'm forwarding three of the responses that the announcement post on Mastodon received:

---

If I can give a suggestion: don't block the whole UI while loading the wasm, but just the "convert" button, so that people can pick their document and options in parallel to the code loading

---

#pandoc is one of the world wonders! Thank you!

---

Pandoc is about as close as civilisation has come to magic so far. (ffmpeg is also close.)

[jgm]

Good suggestion on the blocking. I will look into that. [DONE!]

[tarleb]

One more:

@pandoc incredible work! I'll definitely be using this extensively :)

May I report a bug? It seems like files with names in Arabic (or other Unicode non-ASCII characters) gives this error: ERROR: Unexpected end-of-input while parsing string literal

Changing the file name to ASCII characters fixes the problem.

Thanks again for your work 🙏

[jgm]

Added #11447 to track this

[jgm]

I had Claude diagnose it; the problem, it suggests, is that the JavaScript string length is used to allocate space for the UTF-8 encoded string, which leads to not enough space being allocated when you have code points that need 3 or 4 bytes.

[jgm]

OK, it seems to be fixed now. Let me know if there are more issues.