workflow.md

SpaCy Chinese model training workflow

get preprocessed Chinese Wikipedia corpus

see project chinese-wikipedia-corpus-creator for more details.

produce wikipedia corpus

• input: -
• output: token_cleaned_plain_files/
• script: create_wikipedia_corpus.bash

copy corpus to workspace

• input: `chinese-wikipedia-corpus-creator/token_cleaned_plain_files/``
• output: token_cleaned_plain_files/
• script: move_wikipedia_corpus.bash

computing word frequency

• input: token_cleaned_plain_files/*
• output: WORDS_FREQ.txt
• script: compute_words_freq.bash

merge all files into one

• input: token_cleaned_plain_files/*
• output: WORDS.txt
• script: merge_all_text_files.bash

compute brown cluster

brown cluster computing software

Official software is brown-cluster.

install

• input: -
• output: ``
• script: download_and_compile_brown_cluster.bash

computing

• input: WORDS.txt
• output: WORDS-c1000-p1.out/*
• script: compute_brown_cluster.bash

compute word vector

• input: token_cleaned_plain_files/*
• output: WORDS_VECS.txt
• script: compute_plain_word_vec.bash

initial SpaCy model

build base model

• input: ./WORDS-c1000-p1.out/paths WORDS_VECS.txt WORDS_FREQ.txt
• output: spacy_models/base_model/**/*
• script: create_init_model.bash

modify model name

• input: spacy_models/base_model/meta.json
• output: spacy_models/base_model/meta.json
• script: update_model_meta.py

getting UD_Chinese-GSD corpus

download

• input: -
• output: corpus/UD_Chinese-GS.zip
• script: download_UD_Chinese-GSD_corpus.bash

extracting

• input: corpus/UD_Chinese-GSd.zip
• output: corpus/UD_Chinese-GSd
• script: extract_UD_Chinese-GSD_corpus.bash

convert to simplified Chinese

• input: corpus/UD_Chinese-GSd/zh-ud-*.conllu
• output: corpus/UD_Chinese-GSd/zh-simplified-ud-*.conllu
• script: convert_UD_Chinese-GSD_corpus.bash

convert UD corpus format

• input: .corpus/UD_Chinese-GSD/zh-simplified-ud-*.conllu
• output: corpus/spacy/zh-simplified-ud-*.conllu
• script: format_convertor.bash

init spacy model with word vector & word cluster & word frquence

• input: WORDS_FREQ.txt, WORDS-c1000-p1.out/paths, WORDS_VECS.txt
• output: zh_model/*
• script: init_model.bash

train SpaCy model for POS and dependency parser

• input: zh_model corpus/spacy/zh-simplified-ud-*.conllu
• output: dependency_model
• script: train_model.bash

translate onotNote 5 to spacy json file

• input: TODO
• output: TODO
• script: onto_to_spacy_json.bash

train SpaCy model for NER parser

• input: zh_model china_ner_train.json china_ner_eval.json
• output: ner_model
• script: train_ner.bash

merge sub-model

• input: spacy_models/dependency_model, spacy_models/ner_model
• output: spacy_models/final_model
• script: merge_submodel.py

create package

• input: spacy_models/final_model/
• output: spacy_package/
• script: ./create_model_package.bash