Remote Display proof-of-concept

[ewired]

A Remote Display application for Plato, tested on Clara 2E and Libra 2. An MQTT server relays messages between the Kobo device and Firefox extension. The extension controls the browser with gestures, and encodes tab screenshots as display updates using a WebAssembly module. Display update keyframes are QOI images, or lossy JPEG as fallback if any display update is too large. Interframes are provided by a bidiff of the previous and new frame for QOI keyframes only. All display updates are compressed with zstd.

The first iteration of Remote Display used zlib compressed PBM or PGM images, which could not be as efficiently diffed for interframes. 1-bit PBM+zlib frames had a great payload size but sacrificed grays. Then, JPEG XL was used instead to allow user-configured lossiness/quality, with lossless having better compression than PGM+zlib. 1-bit images compressed worse than 8-bit grayscale with lossless JPEG XL. Lossy JPEG XL didn't improve payload sizes much over lossless. It also had very slow encode and decode times in either mode, lossy being the slowest, made worse by instantiating ImageMagick in WASM for every frame. A custom WASM module to improve JPEG XL performance and use interframes was planned, but there is currently no pure-Rust encoder supporting the interframe feature easily.

QOI+zstd compresses as well as lossless JPEG XL for reading most webpages, and a fallback to lossy JPEG for pages with high-detail images prevents oversized MQTT payloads. Bidiff+zstd of QOI for interframes achieves significant or near total reduction in payload size when scrolling or interacting with webpages with no noticeable speed impact, and only one full QOI frame of memory consumption.

Currently the gestures are:

• Swipe north/south to scroll element under start position or viewport by swipe amount
• Back/forward device buttons to scroll up/down at center of viewport by 50%
• Swipe east/west >50% of width to switch to previous/next tab in window
• Swipe east/west <50% of width to focus the previous/next window
• Arrow east/west to navigate tab forward/back
• Arrow north to close current tab
• Arrow south is handled by device, closes remote display app
• Pinch or spread to zoom by 10%
• Corner northeast to scale up image output & scale down browser window by 10% (northwest for opposite)
• Corner southeast/southwest to increase/decrease contrast filter by 25% (removes filter at lowest bound 100%)
• Corner southeast with contrast at 100% to toggle invert filter
• Short hold finger to manually request updated image, resize the window to desired scale, and full update the display
• Long hold finger to open link under finger in new background tab
• Tap to send click event to element under finger
• Draw circle clockwise around held pivot point to refresh page
• Draw circle counterclockwise around pivot to reopen closed tab