LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts

Installation • Quick Start • Training • Evaluation • Citation

Overview

LM-Lexicon is a research framework for Definition Modeling — generating contextual word definitions using LLMs. It supports inference and training across multiple backends with comprehensive evaluation.

Tasks: Word Interpretation · Context Synthesis · Definition Synthesis · Word Sense Disambiguation

Features: Multi-LLM support (GPT, Claude, Gemini, Llama, Qwen) · Mixture of Experts · BLEU/ROUGE/BERTScore/MoverScore/MAUVE evaluation · DeepSpeed/FSDP/LoRA training · Few-shot ICL

Installation

# Clone and setup
git clone https://github.com/your-username/LM-Lexicon.git
cd LM-Lexicon
conda create -n lmlexicon python=3.10 && conda activate lmlexicon

# Core dependencies
pip install -r inference/requirements.txt

# Training (optional)
pip install deepspeed peft wandb flash-attn liger-kernel

# Evaluation metrics (optional)
pip install sacrebleu bert-score mauve-text nltk

Requires Python 3.8+, PyTorch 2.0+, CUDA 11.8+ (GPU).

Quick Start

# Local model inference
cd inference/
python main.py \
  --task word-interpretation \
  --model /path/to/llama-3-8b \
  --prompt_path prompts/word-interpretation.txt \
  --input_path dataset/3D-EX/test.jsonl \
  --shot_num 0 --max_tokens 64 \
  --run_local_model --evaluate

# API model inference
python main.py \
  --task word-interpretation \
  --model gpt-4-turbo \
  --api_key YOUR_API_KEY \
  --prompt_path prompts/word-interpretation.txt \
  --input_path dataset/3D-EX/test.jsonl \
  --shot_num 3 --evaluate

Training

cd training/

# Single GPU
python finetune.py --model_config run_config/llama_config.json

# Multi-GPU with DeepSpeed
deepspeed --num_gpus=4 finetune.py \
  --model_config run_config/llama_config.json \
  --deepspeed run_config/deepspeed_config_zero2.json

# LoRA fine-tuning
python finetune.py --model_config run_config/llama_config.json --use_lora

Example config (run_config/llama_config.json)

{
    "model_type": "llama-3",
    "model_name_or_path": "/path/to/llama-3-8b",
    "do_train": "True",
    "do_eval": "True",
    "do_predict": "True",
    "data_path_train": "dataset/3D-EX/train.jsonl",
    "data_path_valid": "dataset/3D-EX/valid.jsonl",
    "data_path_test": "dataset/3D-EX/test.jsonl",
    "output_dir": "trained_models/llama-3-8b-3dex",
    "batch_size": 32,
    "per_device_train_batch_size": 2,
    "per_device_eval_batch_size": 4,
    "num_epochs": 3,
    "learning_rate": 2e-5,
    "cutoff_len": 128
}

Datasets

Supported: 3D-EX (RANLP 2023) · WordNet · Oxford · Wikipedia · Urban Dictionary

Input format (JSONL):

{"term": "frozen", "context": "frozen with horror", "definition": "unable to move or act because of fear", "source": "3D-EX"}

Evaluation

Lexical: BLEU, ROUGE-L, METEOR, NIST · Semantic: BERTScore, MoverScore, MAUVE

We adopt the one-to-many evaluation protocol (Huang et al., 2021), reporting the max score across all reference definitions for polysemous words.

Project Structure

LMLexicon/
├── inference/          # Inference (API/local models, prompts, evaluation)
└── training/           # Training (MoE, DeepSpeed, LoRA, configs)

References

1. Ishiwatari et al. (2019) - Learning to Describe Unknown Phrases with Local and Global Contexts
2. Huang et al. (2021) - Definition Modelling for Appropriate Specificity
3. Kong et al. (2022) - Multitasking Framework for Unsupervised Simple Definition Generation
4. Zhang et al. (2022) - Fine-grained Contrastive Learning for Definition Generation
5. Giulianelli et al. (2023) - Interpretable Word Sense Representations via Definition Generation

Citation

@inproceedings{liu2026lmlexicon,
    title = "{LM}-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts",
    author = "Liu, Yang  and
      Yang, Jiaye  and
      Li, Weikang  and
      Liang, Jiahui  and
      Li, Yang  and
      Yan, Lingyong",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.eacl-long.1/",
    doi = "10.18653/v1/2026.eacl-long.1",
    pages = "1--22",
    ISBN = "979-8-89176-380-7",
    abstract = "We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7{\%} BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10{\%} improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1{\%}) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications."
}

License

MIT License - see LICENSE for details.