Optimize image blobs

[willwade]

Optimize Embedded Image Blobs with ARASAAC/OpenSymbols URLs

Summary

Reduces repository size by about.. 50% by replacing embedded image blobs with external ARASAAC/OpenSymbols URLs and applying lossless compression.

Problem

• 1,274 embedded image blobs across 39 communicator boards
• ~375 MB of embedded blobs (51.7% of repository)
• Many blobs are duplicates of publicly available symbols
• Increases clone time and storage requirements

Solution

Implemented 4 things!

1. 1 - Exact Matching (SHA256): 444 blobs → ARASAAC/OpenSymbols URLs
2. 2 - Fuzzy Label Matching: 4,547 symbols matched (88-92% confidence) (<- This is the sketchiest part!)
3. 3 - Perceptual Hashing: 555 additional blobs migrated (<- Second sketchiest part)
4. 4 - Lossless Compression: 20% reduction on remaining blobs ( Easy win)

Results

• 444 blobs migrated (Layer 1)
• 555 additional blobs migrated (Layer 3)
• 830 custom artwork images preserved as blobs
• 193.99 MB saved (51.7% reduction) (BUT Note: a lot of this is just from converting from OBZ to json and it strips out the embedded images)
• 67 files modified across 39 boards

Files Changed

• 67 communicator board files (.grd.json)
• Metadata files (live_metadata.json, live_predefined_*.json)
• 25 primary communicator boards

Testing

I've tried to manually test all changes by loading the pages up. BUT I will be honest - I cant be 10000% sure we havent messed something up. If we could A/B test this.. Or get someone to test this out that would be fab

Notes

• Separated from vocal flair changes (fix-remaining-openboards PR )
• Should be merged AFTER fix-remaining-openboards is done..

I should do a blogpost on this - as I learnt a ton about hashing. Ideally opensymbols and arasaac should create hashes on their API - that would be handy

[klues]

Awesome work!
As said, I won't merge this because it changes all default gridsets and I don't want to change gridsets from cooperation partners without confirmation from them that the changes are fine and don't mess up something.

However for these sets I think we could merge your improvements:

• all from the openboardformat page: https://www.openboardformat.org/examples
  • including the vocal flair 84 - which you didn't include in the other PR (and maybe others from openboardformat which we do not have right now)
• AsTeRICS Grid default: because I know it by heart and will be able to check quickly if everything is still working

Question:

• which format are the lossless compressed images? Still the original one, or some new/fancy format (which maybe isn't supported by all browsers)?

If you create a new PR changing only the sets I've mentioned above, please do the first PR with only running npm run generate-beta which only creates the metadata files for the beta environment at grid.asterics.eu/latest. There I can check the changes before merging the changes generated with npm run generate which affects the files for the prod version.

[willwade]

yeah - look at the previous PR - that should be good for JUST the vocal flair stuff #9 - I've removed all the grids in ->this<- pr here..

re: Formats. No - we could of swapped to WebP but I didnt want to do that. Basically just recompressed with guetzli for png jpeg ..

so to confirm: Look at the previous PR first - check that ok - then if good we can review this together a bit more..

[klues]

I've already merged #9 - but it only contained the vocal flair 112 not the 84 anymore. Vocal flair 112 is already available in prod - with 3.5MB instead of 60MB - which is great!

Basically just recompressed with guetzli for png jpeg ..

ok, great!