scripts

Running Scripts

Before running, please define EXEHOME, OUTPUTHOME, and DATAHOME accordingly in the script.

• e.g.,

  EXEHOME=/home/username/SelfEval-Guided-Decoding/src
  DATAHOME=/home/username/SelfEval-Guided-Decoding/data
  OUTPUTHOME=/home/username/SelfEval-Guided-Decoding/outputs/${dtname}/${split}_outputs

We provide three types of example scripts as follows: (1) baseline running; (2) ours running; (3) LLM evaluating.

PS: please adjust the variables dtname and split to specify the dataset

Baseline Running

(main code: src/generate_code_baseline.py)

• arithmetic reasoning -- run_baseline.sh

• symbolic reasoning -- run_baseline_symbolic.sh

• commonsense reasoning -- run_baseline_commonsense.sh

Ours Running

(main code: src/generate_code.py)

• arithmetic reasoning

  • GSM8K: Ours (PAL), Ours (CoT)
  • AQUA: Ours (PAL)
  • SVAMP: Ours (PAL)
  • ASDiv: Ours (PAL)
  • TabMWP: Ours (PAL)

• symbolic reasoning

  • Date Understanding: Ours (PAL)
  • Object Counting: Ours (PAL)

• commonsense reasoning

  • CSQA: Ours (CoT)
  • StrategyQA: Ours (CoT)
  • Sports Understanding: Ours (CoT)

LLM Evaluation

(main code: src/self_evaluate_code.py)

Run run_self_evaluation.sh