The anonymization process is implemented to prevent the exposure of confidential information in the datasets. This is crucial for ensuring that sensitive details such as personal names, identification numbers, and phone numbers do not become part of the training data and, subsequently, the output generated by the language models.
To achieve this, the document describes using an in-house Named Entity Recognition (NER) model specifically trained on the LST-20 Corpus. The NER model helps in identifying and filtering out names and other identifiable information from the dataset. This process helps ensure that the final model adheres to privacy standards and protects individual privacy.
cd src/scripts/anonymization
python blind.py --config_filename=config/blind_pdpa.yamlThis code will read the datasets, process, and save the output in jsonl format.