o i@szdZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m ZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4e45e6Z7dZ8dZ9dZ:dZ;dZe3e*Gddde-Z?e?Z@dS)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer cached_file)convert_gguf_tokenizer)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )r r WordLevel WordPiece)tokenizer_file vocab_filec)s:eZdZdZeZdZdZedmddZ fddZ e de fd d Z e de fd d Zdnd ededBdeefddZddZe ddZe ddZejddZejddZddZe defddZdeeeffddZe deeeffdd Ze deeeffd!d"Ze deeeffd#d$ZeZeZ deeeffd%d&Z!de fd'd(Z"defd)d*Z#e de$fd+d,Z%e de&fd-d.Z'       /dod0e(d1e dBd2e dBd3e d4e d5e d6e d7e deeee)fe*e(ffd8d9Z+d:edefd;d<Z,d=ededBfd>d?Z-dmd@e*eeBdefdAdBZ.dmdCe defdDdEZ/dmdFee*eBdGe dee*eBfdHdIZ0dpdJedCedBdKe de*efdLdMZ1dNe2dOe3dPedQedRedBdSedBf dTdUZ4dd/e2j5e3j6ddVddddddddddd/dfdJe7e8Be*e7Be*e8BdWe7e8Be*e7Be*e8BdBdKe dNe2dOe3dPedBdQedXe dRedBdSedBdYe dBd1e dBd2e dBd3e d4e d5e d6e d7e dZe dBde9f(d[d\Z:d]e*edefd^d_Z;  dqd`ee*eBdGe dae dBdefdbdcZ<  drd ee=j>Bddeedefdfe dBdedBdeedeff dgdhZ?   dsdidjZ@e       dtdkdlZAZBS)uTokenizersBackendaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc st|}|dd}|dur(tj|r(|tusd|jvs|r(t||d<|S|durtj|rt|j }t |dd }t |}Wdn1sNwY| di dd}|jdurot|trnttt|}nC|jjd kr|rt|d ttfrd d |D}n*|jjd krddt|D}n|jjdks|jjdkrt|trddt|D}||d<t|dd} d| divr| r| jdkr|dd} dd | D} | |d<|dur||d<|S| d} | d} | d}| d} t| tr#| dr#tj| r#ddlm} | | d| \|d<|d<|St| trtj| r| drzhddlm}|| j|jfi|}z!ddlm}| |j}|durit|dri|jd0i|}Wn t y}zt!"d |jd!|d"WYd}~nd}~wwt|d#r|j#d0i|}W|SW|St y}z+t!"d$| d%|d&dd'lm$}|| | d(d)| \|d<|d<WYd}~|Sd}~ww|durt| trtj| r| |d<|d}| dur t| tr tj| r | |d<|d} | durX|jdurX|jjdkrXt|trXd*t%t&d+ttffd,d- gd.}t'}|D]}||vrL|(||gq;t)||d/} | |d<|S)1zs Build a `tokenizers.Tokenizer` backend from the available serialization files (tokenizer.json, sentencepiece models, tekken.json, vocab/merges). r#N__init__tokenizer_objectutf-8encodingmodelvocabr rcSg|]}t|qS)tuple).0itemr.r.l/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/tokenization_utils_tokenizers.py z>TokenizersBackend.convert_to_native_format..r!cSsi|]\}}||qSr.r.r0itokenr.r.r2 z>TokenizersBackend.convert_to_native_format..r r"cSs(i|]\}}t|tr|dn||qSr) isinstancelistr5r.r.r2r8s(mergescSs,g|]}t|trt|dnt|qS) )r;strr/split)r0merger.r.r2r3s,post_processorr$ merges_filez tekken.jsonr)MistralConverter)r$.model)SentencePieceExtractor)SLOW_TO_FAST_CONVERTERSconvert_from_spmz,Could not reorder vocab using converter for z due to z/. Falling back to raw SentencePiece extraction.convert_from_spm_modelz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.)TikTokenConverterextra_special_tokens)r$rKvaluesreturncsHg}|D]}|dur qt|ttfr||q|t|q|SN)r;r<r/extendappendr?)rL collectedval_iter_special_tokensr.r2rTszHTokenizersBackend.convert_to_native_format.._iter_special_tokens) pad_token unk_token bos_token eos_token sep_token cls_token mask_tokenadditional_special_tokensrK) skip_tokensr.)*dictpopospathisfiler%__dict__ TokenizerFast from_filerBopenjsonloadgetr+r;r<mapr/__name__ enumerategetattrr?endswithconvert_slow_tokenizerrDextract_vocab_merges_from_modelrFextractrGhasattrrH ExceptionloggerwarningrIrJrrsetupdater)clstrust_remote_codekwargs local_kwargsfast_tokenizer_file processortokenizer_handletokenizer_jsonr, model_typer=r$rCrDrFrGconverter_classerJspecial_tokens_keysr]keyr.rSr2convert_to_native_formatds               & &        $$0    z*TokenizersBackend.convert_to_native_formatcs|dd}|dd}|dd}|di}|dd}|d}|d} |d } d} |dur9t|} n|durItj|rIt|} n|durt |d d |fi|} t | } | d d }| d}| d}t ||\} }| |t |dkr| |n^|jdur| dur| durt| tr| nddt| D}tt|| ddd} n6t| trtt| gddd} n&t| tr| rt| dttfrtt| |ddd} n |jdurtd|dur|dur|jdur|dd|dd| dur| |_|jdur td|jj}|dur@|jjd.T)r,r=fuse_unkdropoutunk_id)r,ra9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.rWzrXzz3The backend tokenizer is not correctly initialized. max_lengthtruncation_side directionstridetruncation_strategystrategyrUpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofbackend tokenizers add_bos_token add_eos_tokenrBcSsh|]}tt|qSr.hashreprr0r7r.r.r2 Kz-TokenizersBackend.__init__..cs$g|]\}}tt|vr|qSr.r)r0indexr7added_tokens_decoder_hashr.r2r3Ls z.TokenizersBackend.__init__..cS|dSNrr.)xr.r.r2Nz,TokenizersBackend.__init__..rcSr-r.r?rr.r.r2r3Qr4cSsg|]}|rt|qSr.r)r0tr.r.r2r3br)speciali pre_tokenizerfix_mistral_regex) init_kwargsrr.):r_ricopydeepcopyr`rarbrdrerrrrwlen _tokenizerr;r^rlr r<r/r ValueError setdefault truncationenable_truncation no_truncationpaddingenable_padding_add_bos_token_add_eos_tokenrB_should_update_post_processorsuperr&r$rsplit_special_tokensencode_special_tokensrsorteditemsadded_tokens_encoderkeys_special_tokens_maprLr?rP_extra_special_tokensrr add_tokensget_vocab_sizeNotImplementedErrorrm_patch_mistral_regexrupdate_post_processor)selfargsrzr'rr|rrr$r,r=fast_tokenizer gguf_path gguf_param architecturetokenizer_dictradditional_kwargs vocab_dict _truncation_paddingexplicit_bos_eos_in_kwargsrB tokens_to_addencoderspecial_token_valuer7tokensall_named_tokens vocab_size __class__rr2r&s                                        zTokenizersBackend.__init__rMcCdS)NTr.rr.r.r2is_fastszTokenizersBackend.is_fastcCs@d|jvr|jddrt|dr|jrtj|jSdSdS)z `bool`: Whether or not the slow tokenizer can be saved. For a sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. r$rEFT)vocab_files_namesrnrrr$r`rarbrr.r.r2can_save_slow_tokenizers z)TokenizersBackend.can_save_slow_tokenizersave_directoryfilename_prefixcCsptj|std|ddStj||r|dndtd}tj|jtj|kr5t |j||fS)NzVocabulary path (z) should be a directory-rr$) r`raisdirrterrorjoinVOCAB_FILES_NAMESabspathr$r)rrrout_vocab_filer.r.r2save_vocabularys  z!TokenizersBackend.save_vocabularycCs|j}|j}|dur|jrtd|j}|j}|dur#|jr#d|_dS|jr*|dndd|jr6d|dnd}||jrEd|d ndd |jrQd|d nd}g}|jra|||f|jrk|||ftj |||d |j _ dS) ze Updates the underlying post processor with the current `bos_token` and `eos_token`. Nz)add_bos_token = True but bos_token = NoneFz:0 rz$A:0r>z:0z:1z $B:1)singlepairspecial_tokens) rW bos_token_idrrrX eos_token_idrrPrTemplateProcessingrrB)rbosreosrrrrr.r.r2rs&.6z'TokenizersBackend.update_post_processorcC t|ddS)NrFrmrr.r.r2r zTokenizersBackend.add_eos_tokencCr)NrFrrr.r.r2rrzTokenizersBackend.add_bos_tokencCt|d||dS)Nrobject __setattr__rrvaluer.r.r2r cCr)Nrrrr.r.r2rrcCsg}|jD]!}|durqt|tr||qt|tr(|t|dddq|jD]}t|tr9||q,t|trH|t|dddq,|rR|j|ddt|dds^|j j durd| dSdS)a[ Post-initialization hook that runs after the tokenizer is fully set up. This is called by from_pretrained() after loading the tokenizer, which allows us to add any special tokens that may have been passed as AddedToken objects. Child classes should call super()._post_init() if they override this method. NTF)r normalized)rr) rrLr;rrPr?rrrmrrBr)rr token_valuer7r.r.r2 _post_inits(        zTokenizersBackend._post_initcC|jjddS)zP `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrrrr.r.r2rszTokenizersBackend.vocab_sizecCr )NTr )r get_vocabrr.r.r2rszTokenizersBackend.get_vocabcCs|SrN)rrr.r.r2r,szTokenizersBackend.vocabcC ddt|jdddDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cSi|]\}}|j|qSr.contentr0vkr.r.r2r8rz:TokenizersBackend.added_tokens_encoder..cSrrr.r1r.r.r2rrz8TokenizersBackend.added_tokens_encoder..rrrrrr.r.r2rs z&TokenizersBackend.added_tokens_encodercCs |jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderrr.r.r2r s z&TokenizersBackend.added_tokens_decodercCr)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. cSrr.rrr.r.r2r8 rz5TokenizersBackend.get_added_vocab..cSrrr.rr.r.r2r rz3TokenizersBackend.get_added_vocab..rrrr.r.r2get_added_vocabs z!TokenizersBackend.get_added_vocabcCr)zN Returns True, to avoid expensive `assert tokenizer` gotchas. Tr.rr.r.r2__bool__"szTokenizersBackend.__bool__cCr )zD Size of the full vocabulary with the added tokens. Tr rrr.r.r2__len__(szTokenizersBackend.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rrr.r.r2backend_tokenizer.sz#TokenizersBackend.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderrr.r.r2r5szTokenizersBackend.decoderTr*return_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec Cs|dur d|jv}|durd|jv}|r |jdur |g|j} n|g} tt} | D]>} | d| j|r=| d| j|rG| d| j|rQ| d| j|r[| d| j |rg| dt | jq)| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) Ntoken_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrr<rPidstype_idsr(r*offsetsr) rr*r r!r"r#r$r%r& encodings encoding_dictrr.r.r2_convert_encoding<s,  z#TokenizersBackend._convert_encodingr7cCs|j|}|dur |jS|SrN)r token_to_id unk_token_id)rr7rr.r.r2#_convert_token_to_id_with_added_vocks z5TokenizersBackend._convert_token_to_id_with_added_vocrcCs|jt|SrN)r id_to_tokenint)rrr.r.r2_convert_id_to_tokenqsz&TokenizersBackend._convert_id_to_token new_tokenscCs|r|j|S|j|SrN)radd_special_tokensr)rr:rr.r.r2 _add_tokensts  zTokenizersBackend._add_tokensrcCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rrr.r.r2r=zs z+TokenizersBackend.num_special_tokens_to_addr.skip_special_tokenscCs`t|tr |j|Sg}|rt|jnt}|D]}t|}||vr$q||j|q|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )r;r8rr7rvall_special_idsrP)rr.r>r ids_to_skiprr.r.r2convert_ids_to_tokenss  z'TokenizersBackend.convert_ids_to_tokenstextr;cKs|jd|||d|S)N)rB text_pairr;r.) _encode_plusr)rrBrr;rzr.r.r2tokenizeszTokenizersBackend.tokenizepadding_strategyrrrrrc s|jj|jj}|tjkrdur|jn&|||j|jd}dur'd} n fdd|D} | |kr=|jjdi||t j krO|durM|j dSdS|t j krV|nd} | |dur_|n|j |j|j|j|d}||krz|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)rrrrcsi|] }||dqSrNri)r0rrr.r2r8rz@TokenizersBackend.set_truncation_and_padding..)rrpad_idrUrrr.)rrrrDO_NOT_TRUNCATErrrrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idrUrr) rrFrrrrrrtargetcurrentrr.rHr2set_truncation_and_paddings>!    z,TokenizersBackend.set_truncation_and_paddingrrCis_split_into_wordsreturn_tensorsrc# sHdd}||s td|dur||std|r-t|ttfo+|o+t|dttf}nt|ttf}|rht|tr?td|durZt|t|krZtdt|dt|d|durett||n|}n |ro||fgn|g}t|ttfstd t|d j ||||| | d |durj }j j |kr|j _ j j |||d }fd d|D}i}|ddD]fdd|D}||<qdd|D}rg}t|D]\}\}} ||gt|d7}q||d<|dD] }!|!|qt||| d}"|s"| dur"s"tdd|"D|"j}"|"S)NcSst|trdSt|ttfr_t|dkrdSt|dtrdSt|dttfr]t|ddks9t|ddtr;dSt|ddttfr[t|dddkpZt|dddtSdSdSdS)NTrF)r;r?r<r/r)rr.r.r2_is_valid_text_inputs  "*z._is_valid_text_inputztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rFrrrrr)r;is_pretokenizedc s&g|]}j|dqS))r*r r!r"r#r$r%r&)r3)r0r*)r!r%r$r"r#r rr&r.r2r3ds z2TokenizersBackend._encode_plus..cs"g|] \}}|D]}|q qSr.r.)r0r1rrrr.r2r3us"cSsg|] \}}|D]}|qqSr.r.)r0rr1rr.r.r2r3wsr)overflow_to_sample_mapping) tensor_typecSs8i|]\}}|t|dkrt|dtr|dn|qSr:)rr;r<)r0rrr.r.r2r8s&z2TokenizersBackend._encode_plus..)rr;r<r/r? TypeErrorrziptyperQrrr encode_batchrl&_eventual_warn_about_too_long_sequencerrr1)#rrBrCr;rFrrrrRrrrSr r!r"r#r$r%r&rrzrT is_batchedbatch_text_or_text_pairsr1tokens_and_encodingssanitized_tokensstacksanitized_encodingsrXr6toksrr)batched_outputr.) rr!r%r$r"r#r rr&r2rDs&       zTokenizersBackend._encode_plusrcCs$|jjdur |jj|Sd|S)Nr>)rrdecoder)rrr.r.r2convert_tokens_to_strings  z*TokenizersBackend.convert_tokens_to_string token_idsclean_up_tokenization_spacescKs|ddt|tr|g}t|tr|d}|jj||d}|dur%|n|j}|ret|dr;t|j r;| |}|S| dd dd d d  d d  d d dd dd dd dd dd}|S)Nuse_source_tokenizerr))r>clean_up_tokenizationz .rUz ??z !!z ,,z ' 'z n'tzn'tz 'mz'mz 'sz'sz 'vez'vez 'rez're) r_r;r8r^rrgrjrrcallablerlreplace)rrir>rjrzrBr.r.r2_decodes4      zTokenizersBackend._decode file_names. legacy_formatcCs@t|}tj||r|dndt}|j|||f}|S)Nrr)r?r`rarTOKENIZER_FILErsave)rrrtrurr#r.r.r2_save_pretraineds  z"TokenizersBackend._save_pretrainedc st|j}|d}|d} d} |dddkr)i|dd<g|dd<nW|ddd kre|dd durd|dd } |dd| d } durU| vrU| } d |dd <| d gg|dd<n|ddd vrti|dd<n td|ddddurd|dvr|ddvr|dd|dd<tt|g} |D]5} | dd}| dd}|ddd kr|sqdur| dvrՈ| d| d<| t d,i| q|dur| ||dddkrd|vr|dddur|dd|d<|dddkr'd|vr'|dddur'|dd|d<|ddd kr9| dur9| |d<|ddurn|dddksg|dddkrnd|dvrnt dd|ddDrnt j|d<t|dd}|d,|| d|}j|||d| dur!t}d| vr| dD]D}| d|d }durfd!d"|D}|| d|d <|D]}|}|durtd#qfd$d"|D| d|d%<qd&D]0}|| vr| |\}}dur|vr|}|}|dur td#||g| |<q| |d<tt||j}tjD]A}t||durit||}durF|vrF|}|j|d}t|t ret ||j|j|j|jd'd(||<q)|||<q)|jrt|jng}|dur| |t |d kr||d)<|d*<z |j!d,i|WSt"y}z!d+t#|vr|d*d|j!d,i|}|_|WYd}~Sd}~ww)-uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokensrBNr+r\r r,r=r rrg)r!r"z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.rVridrcontinuing_subword_prefixend_of_word_suffixr ByteLevelSequence pretokenizerscss|] }|ddkVqdS)r\r}Nr.)r0 pretokenizerr.r.r2 ;s  z.initial_alphabet)rr)rtrainerrrcsg|]}||qSr.rGr)special_tokens_mapr.r2r3Mrz=TokenizersBackend.train_new_from_iterator..zQAttempted to set a token in the post processor that does not exist in the mappingcsg|]}|qSr.)r4r)rr.r2r3Vr9r.)rxsepT) single_wordlstriprstripr rrKr'z7multiple values for keyword argument 'tokenizer_object'r.)$rgloadsrto_strr_rrdfrom_strdumpsrPrrOanypre_tokenizers_fastr}alphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorr4rrrSPECIAL_TOKENS_ATTRIBUTESrmrrir;rrrr rKrrrZr?)r text_iteratorrrnew_special_tokensrrzrryrBrVrr added_tokenrr trainer_classrtrained_tokenizer_jsonrrr7token_id special_tokenspecial_token_fullrKr new_tokenizerr.)rrr2train_new_from_iterators"                  "             z)TokenizersBackend.train_new_from_iteratorc  s0ddlddlmddlm} ddlm} dtdtffdd } t r'd }|dur|s6|s| |r| |d |||d d |d }d }|durt |dd }t |}Wdn1s^wY| d}| d}|r| || dkr|r|dur|dvr|Sn|r| || dkr|Sd }|s|s| |r|rd|vrt|d|d| durt|dd st|dd td|d|S| d ust|dd rt|dd ddl}|jj|ddd}|jj}t||jjr||jjd<|St||jjr |jjd d d}|j||g|j_|S)af Patches mistral related tokenizers with incorrect regex if detected 1) Local file with an associated config saved next to it >> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` rN) model_info)versionrmodel_idrMcs.|}|jdurdd|jrdSdS)Nzbase_model:.*mistralairTF)tagssearchr)rr+rrer.r2is_base_mistrals  z?TokenizersBackend._patch_mistral_regex..is_base_mistralTz config.jsonF) cache_dirr7local_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr(r)transformers_versionrz4.57.2)mistralmistral3voxtral ministralpixtralz4.57.3rz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)rhuggingface_hubr packagingrtransformers.utils.hubrr?boolrrfrgrhriparsesetattrrmrtrurpre_tokenizersSplitRegexrrr;r~ Metaspacer})rxrpretrained_model_name_or_pathr7rrris_localrrrzrrr _config_filemistral_config_detectedf_configrtransformers_model_typersplit_pretokenizercurrent_pretokenizerr.rr2rs            $  z&TokenizersBackend._patch_mistral_regex)FrN)NNFFFFT)NF)FN)NN)NNN)NNFNFNN)Crk __module__ __qualname____doc__rrr+r classmethodrr&propertyrrrr?r/rrrrsetterr r8rr^rr,rrr_added_tokens_encoder_added_tokens_decoderrrrrdr DecoderFastr EncodingFastrr<r3r6r9r<r=rArErrrQrKrJrrrrDrhrsr`PathLikerxrr __classcell__r.r.rr2r%Rsv               /($ N         .    Fr%)Arrrgr` collectionsrcollections.abcrshutilrtypingrtokenizers.pre_tokenizersrrrrrrrr rr rdtokenizers.decodersr rtokenizers.modelsr r tokenizers.trainersrrrrrrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utils_baserrrrrrrutilsrrr get_loggerrkrtrvSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEADDED_TOKENS_FILErrr%PreTrainedTokenizerFastr.r.r.r2s\             $   P