o i@sJddlmZddlmZddlmZddlmZmZdede fddZ d S) )Path)Any)TikTokenConverter)TIKTOKEN_VOCAB_FILETOKENIZER_FILEencoding output_dirc Cst|}|jdd|dt}|t}|jjdddt|}t|}zddlm}ddl m }t |tr>||}||j |Wn t ye}zt|} d| vr\td |td |d }~wwt||j|jd } | |d S) a Converts given `tiktoken` encoding to `PretrainedTokenizerFast` and saves the configuration of converted tokenizer on disk. Args: encoding (`str` or `tiktoken.Encoding`): Tokenizer from `tiktoken` library. If `encoding` is `str`, the tokenizer will be loaded with `tiktoken.get_encoding(encoding)`. output_dir (`str`): Save path for converted tokenizer configuration file. T)exist_oktiktoken)parentsr r) get_encoding)dump_tiktoken_bpeblobfilezY`blobfile` is required to save a `tiktoken` file. Install it with `pip install blobfile`.zY`tiktoken` is required to save a `tiktoken` file. Install it with `pip install tiktoken`.N) vocab_filepatternextra_special_tokens)rmkdirrrparentstrabsoluter r tiktoken.loadr isinstance_mergeable_ranks ImportErrorlower ValueErrorr_pat_str_special_tokens convertedsave) rr save_filetokenizer_filesave_file_absoluteoutput_file_absoluter r e error_msg tokenizerr'd/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/integrations/tiktoken.pyconvert_tiktoken_to_fastsD         r)N) pathlibrtypingr#transformers.convert_slow_tokenizerr*transformers.tokenization_utils_tokenizersrrrr)r'r'r'r(s