Questions about future plans

[BLKSerene]

Hi, thanks for your great work. I have several questions about the future plans of this project.

1. Are there any plans to release KhmerSegmenter on PyPI for more convenient installation (and releasing)?
2. While it is claimed in the README here that KhmerSegmenter has zero external dependencies, the usage notes still ask users to install required dependencies like khmernltk (specified in requirements.txt). Is this an inconsistency or am I getting it wrong?
3. Are there any plans to also support sentence tokenization and part-of-speech tagging (and maybe other NLP tasks) just like khmernltk did?

[Sovichea]

Hello @BLKSerene , thank you for checking the project. To answer your questions:

1. I will look into the possibility soon. As of now I'm still assessing the accuracy of the algorithm and tweaking for user experience.

2. Regarding the requirement for khmernltk, it's primarily used to benchmark my algorithm. It doesn't serve any purpose in the calculation of word frequencies.

3. POS tagging is quite straightforward to implement as I can compare the segmented word against the dictionary. However, I don't have a reliable plaintext format of Khmer dictionary yet. I believe there is Chuon Nath dictionary some where in HuggingFace stored in SQL database. I will need to check it and implement it in the near future.

To sum up, the current Python implementation is only for testing the accuracy of Viterbi algorithm for Khmer word segmentation. It's execution speed is nowhere native implementation because Pyton lacks true parallelism. In the near future, I will write a Python C wrapper to utilize the true multi threading capability and host it in PyPI.