Skip to content

Use FST instead of Trie for consuming less memory#10

Merged
mominul merged 3 commits into
mainfrom
fst
Jun 13, 2025
Merged

Use FST instead of Trie for consuming less memory#10
mominul merged 3 commits into
mainfrom
fst

Conversation

@mominul

@mominul mominul commented Jun 10, 2025

Copy link
Copy Markdown
Member

This makes upodesh to use FST instead of Trie which significantly reduces the amount of memory consumption (from 140MB to 2MB), a 98.57% reduction! 🎉

But the runtime performance gets a hit, but I think we can bear this tradeoff in the long run.

Test Case upodesh (mean µs) upodesh-fst (mean µs)
a ~2.134 ~3.341
arO ~6.971 ~11.840
bistari ~4.628 ~9.734

But the fst version of upodesh is still significantly faster than the regex-based implementation!

Test Case upodesh-fst (µs) regex (µs) Difference
a ~3.341 ~194.34 🚀 fst is ~58× faster
arO ~11.840 ~246.53 🚀 fst is ~20.8× faster
bistari ~9.734 ~353.74 🚀 fst is ~36.3× faster

cc @gulshan @mugli

Closes #3

@mugli

mugli commented Jun 10, 2025

Copy link
Copy Markdown

OMG! 😍

@mominul mominul merged commit 5bfd735 into main Jun 13, 2025
2 checks passed
@mominul mominul deleted the fst branch June 13, 2025 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Huge amount of memory allocation

2 participants