o i@s4ddlmZes devrddlmZnddlZzddlZWn e y*ddlZYnwddZ dd Z d d Z d d Z GdddeZGdddeZeeGdddeZeeGdddeZeeGdddeZeeddZddZGdddeZeeGdddeZeed d!Zddl Z ddl!Z!ddlZddl"Z"ddl#Z$dd"l%m&Z&dd#l%m'Z'd$d%Z(d&d'Z)ej*Z+ej*Z,e-ed(ej.e-ed(ej.ej/e_0ej1e_2d)D]Z3e)ee3qe(ee(ee(eeZ4eZ5dd*l6m7Z7ee"j89e:e$j;)this__repr__ __builtin__ Exception __class__ __module____name__)selfstrthisrX/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/sentencepiece/__init__.py _swig_reprs rcfdd}|S)Ncsj|dkr |||dS|dkr|j|dSt||r/ttt||tr/|||dStd|)Nrthisownz(You cannot add instance attributes to %s)rownhasattr isinstancegetattrtypepropertyAttributeErrorrnamevaluesetrrset_instance_attrs zE_swig_setattr_nondynamic_instance_variable..set_instance_attrr)r r!rrr*_swig_setattr_nondynamic_instance_variables  r"cr)Ncs6t||rtt||ts|||dStd|)Nz%You cannot add class attributes to %s)rrrrr)clsrrrrrset_class_attr)s z?_swig_setattr_nondynamic_class_variable..set_class_attrr)r r$rrr'_swig_setattr_nondynamic_class_variable(s r%cr)zlClass decorator for adding a metaclass to a SWIG wrapped class - a slimmed down version of six.add_metaclasscs|j|j|jSN)r __bases____dict__copy)r# metaclassrrwrapper3z$_swig_add_metaclass..wrapperr)r+r,rr*r_swig_add_metaclass1s r.c@seZdZdZeejZdS)_SwigNonDynamicMetazKMeta class to enforce nondynamic attributes (no new attributes) for a classN)r r __qualname____doc__r%r __setattr__rrrrr/8sr/c@seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZddZee ZeeZee ZeeZee ZeeZeeZddZddZddZeZdS)1ImmutableSentencePieceText_ImmutableSentencePiececC |jSr&rrxrrr> z:ImmutableSentencePieceText_ImmutableSentencePiece.cC |j|Sr&r5r7vrrrr8> The membership flagdoccCt|tdSr&)r:ImmutableSentencePieceText_ImmutableSentencePiece_swiginit5new_ImmutableSentencePieceText_ImmutableSentencePiecerrrr__init__Az:ImmutableSentencePieceText_ImmutableSentencePiece.__init__cC t|Sr&)r8ImmutableSentencePieceText_ImmutableSentencePiece__piecerDrrr_pieceE z8ImmutableSentencePieceText_ImmutableSentencePiece._piececCrGr&)r:ImmutableSentencePieceText_ImmutableSentencePiece__surfacerDrrr_surfaceHrJz:ImmutableSentencePieceText_ImmutableSentencePiece._surfacecCrGr&)r5ImmutableSentencePieceText_ImmutableSentencePiece__idrDrrr_idKrJz5ImmutableSentencePieceText_ImmutableSentencePiece._idcCrGr&)r8ImmutableSentencePieceText_ImmutableSentencePiece__beginrDrrr_beginNrJz8ImmutableSentencePieceText_ImmutableSentencePiece._begincCrGr&)r6ImmutableSentencePieceText_ImmutableSentencePiece__endrDrrr_endQrJz6ImmutableSentencePieceText_ImmutableSentencePiece._endcCrGr&)rCImmutableSentencePieceText_ImmutableSentencePiece__surface_as_bytesrDrrr_surface_as_bytesTrJzCImmutableSentencePieceText_ImmutableSentencePiece._surface_as_bytescCrGr&)rAImmutableSentencePieceText_ImmutableSentencePiece__piece_as_bytesrDrrr_piece_as_bytesWrJzAImmutableSentencePieceText_ImmutableSentencePiece._piece_as_bytescCsd|j|j|j|j|jS)Nz3piece: "{}" id: {} surface: "{}" begin: {} end: {} )formatpieceidsurfacebeginendrDrrr__str__bs z9ImmutableSentencePieceText_ImmutableSentencePiece.__str__cCs<|j|jko|j|jko|j|jko|j|jko|j|jkSr&)rXrYrZr[r\rotherrrr__eq__js<z8ImmutableSentencePieceText_ImmutableSentencePiece.__eq__cCs tt|Sr&)hashstrrDrrr__hash__m z:ImmutableSentencePieceText_ImmutableSentencePiece.__hash__N)r r r0rrrrrEr8delete_ImmutableSentencePieceText_ImmutableSentencePiece__swig_destroy__rIrLrNrPrRrTrVrXpiece_as_bytesrZsurface_as_bytesrYr[r\r]r`rcrrrrr3=s.r3c@seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZee ZeeZeeZGdddZeddZddZddZddZeZdS)ImmutableSentencePieceTextcCr4r&r5r6rrrr8vr9z#ImmutableSentencePieceText.cCr:r&r5r;rrrr8vr=r>r?cCrAr&)r#ImmutableSentencePieceText_swiginitnew_ImmutableSentencePieceTextrDrrrrEyrFz#ImmutableSentencePieceText.__init__cCrGr&)r'ImmutableSentencePieceText__pieces_sizerDrrr _pieces_size}rJz'ImmutableSentencePieceText._pieces_sizecC t||Sr&)r"ImmutableSentencePieceText__piecesrindexrrr_piecesrdz"ImmutableSentencePieceText._piecescCrGr&)r ImmutableSentencePieceText__textrDrrr_textrJz ImmutableSentencePieceText._textcCrGr&)r!ImmutableSentencePieceText__scorerDrrr_scorerJz!ImmutableSentencePieceText._scorecCrGr&)r,ImmutableSentencePieceText_SerializeAsStringrDrrrSerializeAsStringrJz,ImmutableSentencePieceText.SerializeAsStringcCrGr&)r)ImmutableSentencePieceText__text_as_bytesrDrrr_text_as_bytesrJz)ImmutableSentencePieceText._text_as_bytesc@0eZdZddZddZddZddZeZd S) z9ImmutableSentencePieceText.ImmutableSentencePieceIteratorcC||_|j|_dSr&)protormlenrr}rrrrEzBImmutableSentencePieceText.ImmutableSentencePieceIterator.__init__cC|jSr&r~rDrrr__len__zAImmutableSentencePieceText.ImmutableSentencePieceIterator.__len__cjt|trfddtjD|j|j|jS|dkr"|j}|dks+|jkr/tdj |S)Ncg|]}j|qSr)r}rr.0irDrr zYImmutableSentencePieceText.ImmutableSentencePieceIterator.__getitem__..rzpiece index is out of range) rsliceranger~startstopstep IndexErrorr}rrrprrDr __getitem__ (  zEImmutableSentencePieceText.ImmutableSentencePieceIterator.__getitem__cCddd|DS)N cSg|] }dt|qSzpieces {{ {}}}rWrbrr7rrrrzUImmutableSentencePieceText.ImmutableSentencePieceIterator.__str__..joinrDrrrr]rFzAImmutableSentencePieceText.ImmutableSentencePieceIterator.__str__Nr r r0rErrr]rrrrrImmutableSentencePieceIterator  rcCrGr&)rirrDrrrpieces z!ImmutableSentencePieceText.piecescC||kSr&rxr^rrrr`z!ImmutableSentencePieceText.__eq__cC t|Sr&rarxrDrrrrcrdz#ImmutableSentencePieceText.__hash__cCs$d|j|jddd|jDS)Nztext: "{}" score: {} {}rcSrrrrrrrrrz6ImmutableSentencePieceText.__str__..)rWtextscorerrrDrrrr]s  z"ImmutableSentencePieceText.__str__N)r r r0rrrrrEr!delete_ImmutableSentencePieceTextrfrmrrrtrvrxrzr text_as_bytesrrrr`rcr]rrrrrius* ric@seZdZeddddddZeZddZej Z dd Z d d Z d d Z GdddZeddZddZddZddZeZdS)ImmutableNBestSentencePieceTextcCr4r&r5r6rrrr8r9z(ImmutableNBestSentencePieceText.cCr:r&r5r;rrrr8r=r>r?cCrAr&)r(ImmutableNBestSentencePieceText_swiginit#new_ImmutableNBestSentencePieceTextrDrrrrErFz(ImmutableNBestSentencePieceText.__init__cCrGr&)r,ImmutableNBestSentencePieceText__nbests_sizerDrrr _nbests_sizerJz,ImmutableNBestSentencePieceText._nbests_sizecCrnr&)r'ImmutableNBestSentencePieceText__nbestsrprrr_nbestsrdz'ImmutableNBestSentencePieceText._nbestscCrGr&)r1ImmutableNBestSentencePieceText_SerializeAsStringrDrrrrxrJz1ImmutableNBestSentencePieceText.SerializeAsStringc@r{) zBImmutableNBestSentencePieceText.ImmutableSentencePieceTextIteratorcCr|r&)r}rr~rrrrrErzKImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__init__cCrr&rrDrrrrrzJImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__len__cr)Ncrr)r}rrrDrrrrzbImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__getitem__..rznbests index is out of range) rrrr~rrrrr}rrprrDrrrzNImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__getitem__cCr)NrcSrznbests {{ {}}}rrrrrrrz^ImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__str__..rrDrrrr]rFzJImmutableNBestSentencePieceText.ImmutableSentencePieceTextIterator.__str__Nrrrrr"ImmutableSentencePieceTextIteratorrrcCrGr&)rrrDrrrnbestsrz&ImmutableNBestSentencePieceText.nbestscCrr&rr^rrrr`rz&ImmutableNBestSentencePieceText.__eq__cCrr&rrDrrrrcrdz(ImmutableNBestSentencePieceText.__hash__cCsddd|jDS)NrcSrrrrrrrrrz;ImmutableNBestSentencePieceText.__str__..)rrrDrrrr]r-z'ImmutableNBestSentencePieceText.__str__N)r r r0rrrrrEr&delete_ImmutableNBestSentencePieceTextrfrrrxrrr`rcr]rrrrrs rc @seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZddZddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Z d4d5Z!d6d7Z"d8d9Z#d:d;Z$dd?Z&d@dAZ'dBdCZ(dDdEZ)dFdGZ*dHdIZ+dJdKZ,dLdMZ-dNdOZ.dPdQZ/dRdSZ0dTdUZ1dVdWZ2dXdYZ3dZd[Z4d\d]Z5d^d_Z6d`daZ7dbdcZ8dddeZ9dfdgZ:dhdiZ;djdkZdpdqZ?drdsZ@dtduZAdvdwZBdxdxeCdydydydydydzd{dzf d|d}ZD x x x x x x x x xdd~dZEddZFddZGddZHddZIdddZJdddZKdddZLdddZM x x x x x xdddZNdddZOdddZPdddZQdddZR x x x x x x x x xdddZSdddZTdddZUdddZVdddZWeXdxfddZYeXfddZZeXfddZ[dddZ\dddZ]dddZ^dddZ_dddZdddZ`ddZaddZbddZcddZdddZedd„ZfddĄZgdddƄZhdxS)SentencePieceProcessorcCr4r&r5r6rrrr8r9zSentencePieceProcessor.cCr:r&r5r;rrrr8r=r>r?cCrAr&)rSentencePieceProcessor_swiginitnew_SentencePieceProcessorrDrrrrErFzSentencePieceProcessor.__init__cCrnr&)r.SentencePieceProcessor_LoadFromSerializedProtor serializedrrrLoadFromSerializedProtordz.SentencePieceProcessor.LoadFromSerializedProtocCrnr&)r,SentencePieceProcessor_SetEncodeExtraOptionsr extra_optionrrrSetEncodeExtraOptionsrdz,SentencePieceProcessor.SetEncodeExtraOptionscCrnr&)r,SentencePieceProcessor_SetDecodeExtraOptionsrrrrSetDecodeExtraOptionsrdz,SentencePieceProcessor.SetDecodeExtraOptionscCrnr&)r$SentencePieceProcessor_SetVocabulary)r valid_vocabrrr SetVocabularyrdz$SentencePieceProcessor.SetVocabularycCrGr&)r&SentencePieceProcessor_ResetVocabularyrDrrrResetVocabulary rJz&SentencePieceProcessor.ResetVocabularycCt|||Sr&)r%SentencePieceProcessor_LoadVocabulary)rfilename thresholdrrrLoadVocabularyz%SentencePieceProcessor.LoadVocabularycGstj|g|RSr&)r'SentencePieceProcessor_CalculateEntropyrargsrrrCalculateEntropysz'SentencePieceProcessor.CalculateEntropycCrGr&)r#SentencePieceProcessor_GetPieceSizerDrrr GetPieceSizerJz#SentencePieceProcessor.GetPieceSizecCrnr&)r SentencePieceProcessor_PieceToIdrrXrrr PieceToIdrdz SentencePieceProcessor.PieceToIdcCrnr&)r SentencePieceProcessor_IdToPiecerrYrrr IdToPiecerdz SentencePieceProcessor.IdToPiececCrnr&)rSentencePieceProcessor_GetScorerrrrGetScorerdzSentencePieceProcessor.GetScorecCrnr&)r SentencePieceProcessor_IsUnknownrrrr IsUnknown rdz SentencePieceProcessor.IsUnknowncCrnr&)r SentencePieceProcessor_IsControlrrrr IsControl#rdz SentencePieceProcessor.IsControlcCrnr&)rSentencePieceProcessor_IsUnusedrrrrIsUnused&rdzSentencePieceProcessor.IsUnusedcCrnr&)rSentencePieceProcessor_IsByterrrrIsByte)rdzSentencePieceProcessor.IsBytecCrGr&)rSentencePieceProcessor_unk_idrDrrrunk_id,rJzSentencePieceProcessor.unk_idcCrGr&)rSentencePieceProcessor_bos_idrDrrrbos_id/rJzSentencePieceProcessor.bos_idcCrGr&)rSentencePieceProcessor_eos_idrDrrreos_id2rJzSentencePieceProcessor.eos_idcCrGr&)rSentencePieceProcessor_pad_idrDrrrpad_id5rJzSentencePieceProcessor.pad_idcCrGr&)r-SentencePieceProcessor_serialized_model_protorDrrrserialized_model_proto8rJz-SentencePieceProcessor.serialized_model_protocCrnr&)r#SentencePieceProcessor_LoadFromFilerargrrr LoadFromFile;rdz#SentencePieceProcessor.LoadFromFilec Ct||||||||| Sr&)r#SentencePieceProcessor__EncodeAsIds rrenable_sampling nbest_sizealphaadd_bosadd_eosreverseemit_unk_piecerrr _EncodeAsIds>z#SentencePieceProcessor._EncodeAsIdsc Crr&)r&SentencePieceProcessor__EncodeAsPiecesrrrr_EncodeAsPiecesArz&SentencePieceProcessor._EncodeAsPiecesc Crr&)r/SentencePieceProcessor__EncodeAsSerializedProtorrrr_EncodeAsSerializedProtoDrz/SentencePieceProcessor._EncodeAsSerializedProtoc Crr&)r.SentencePieceProcessor__EncodeAsImmutableProtorrrr_EncodeAsImmutableProtoGrz.SentencePieceProcessor._EncodeAsImmutableProtoc Ct|||||||||| Sr&)r(SentencePieceProcessor__EncodeAsIdsBatch rins num_threadsrrrrrrrrrr_EncodeAsIdsBatchJz(SentencePieceProcessor._EncodeAsIdsBatchc Crr&)r+SentencePieceProcessor__EncodeAsPiecesBatchrrrr_EncodeAsPiecesBatchMr z+SentencePieceProcessor._EncodeAsPiecesBatchc Crr&)r4SentencePieceProcessor__EncodeAsSerializedProtoBatchrrrr_EncodeAsSerializedProtoBatchPr z4SentencePieceProcessor._EncodeAsSerializedProtoBatchc Crr&)r3SentencePieceProcessor__EncodeAsImmutableProtoBatchrrrr_EncodeAsImmutableProtoBatchSr z3SentencePieceProcessor._EncodeAsImmutableProtoBatchcCrnr&)r!SentencePieceProcessor__DecodeIdsridsrrr _DecodeIdsVrdz!SentencePieceProcessor._DecodeIdscCrnr&)r(SentencePieceProcessor__DecodeIdsAsBytesrrrr_DecodeIdsAsBytesYrdz(SentencePieceProcessor._DecodeIdsAsBytescCrnr&)r$SentencePieceProcessor__DecodePiecesrrrrr _DecodePieces\rdz$SentencePieceProcessor._DecodePiecescCrnr&)r2SentencePieceProcessor__DecodeIdsAsSerializedProtorrrr_DecodeIdsAsSerializedProto_rdz2SentencePieceProcessor._DecodeIdsAsSerializedProtocCrnr&)r5SentencePieceProcessor__DecodePiecesAsSerializedProtorrrr_DecodePiecesAsSerializedProtobrdz5SentencePieceProcessor._DecodePiecesAsSerializedProtocCrnr&)r1SentencePieceProcessor__DecodeIdsAsImmutableProtorrrr_DecodeIdsAsImmutableProtoerdz1SentencePieceProcessor._DecodeIdsAsImmutableProtocCrnr&)r4SentencePieceProcessor__DecodePiecesAsImmutableProtorrrr_DecodePiecesAsImmutableProtohrdz4SentencePieceProcessor._DecodePiecesAsImmutableProtocCrr&)r&SentencePieceProcessor__DecodeIdsBatchrrr rrr_DecodeIdsBatchkrz&SentencePieceProcessor._DecodeIdsBatchcCrr&)r-SentencePieceProcessor__DecodeIdsAsBytesBatchr$rrr_DecodeIdsAsBytesBatchnrz-SentencePieceProcessor._DecodeIdsAsBytesBatchcCrr&)r7SentencePieceProcessor__DecodeIdsAsSerializedProtoBatchr$rrr _DecodeIdsAsSerializedProtoBatchqrz7SentencePieceProcessor._DecodeIdsAsSerializedProtoBatchcCrr&)r6SentencePieceProcessor__DecodeIdsAsImmutableProtoBatchr$rrr_DecodeIdsAsImmutableProtoBatchtrz6SentencePieceProcessor._DecodeIdsAsImmutableProtoBatchcCrr&)r)SentencePieceProcessor__DecodePiecesBatchr$rrr_DecodePiecesBatchwrz)SentencePieceProcessor._DecodePiecesBatchcCrr&)r:SentencePieceProcessor__DecodePiecesAsSerializedProtoBatchr$rrr#_DecodePiecesAsSerializedProtoBatchzrz:SentencePieceProcessor._DecodePiecesAsSerializedProtoBatchcCrr&)r9SentencePieceProcessor__DecodePiecesAsImmutableProtoBatchr$rrr"_DecodePiecesAsImmutableProtoBatch}rz9SentencePieceProcessor._DecodePiecesAsImmutableProtoBatchc Ct|||||||Sr&)r(SentencePieceProcessor__NBestEncodeAsIdsrrrrrrrrrr_NBestEncodeAsIdsr-z(SentencePieceProcessor._NBestEncodeAsIdsc Cr2r&)r+SentencePieceProcessor__NBestEncodeAsPiecesr4rrr_NBestEncodeAsPiecesr-z+SentencePieceProcessor._NBestEncodeAsPiecesc Cr2r&)r4SentencePieceProcessor__NBestEncodeAsSerializedProtor4rrr_NBestEncodeAsSerializedProtor-z4SentencePieceProcessor._NBestEncodeAsSerializedProtoc Cr2r&)r3SentencePieceProcessor__NBestEncodeAsImmutableProtor4rrr_NBestEncodeAsImmutableProtor-z3SentencePieceProcessor._NBestEncodeAsImmutableProtoc Crr&)r1SentencePieceProcessor__SampleEncodeAndScoreAsIds rr num_samplesrwor include_bestrrrrrrr_SampleEncodeAndScoreAsIdsr z1SentencePieceProcessor._SampleEncodeAndScoreAsIdsc Crr&)r4SentencePieceProcessor__SampleEncodeAndScoreAsPiecesr=rrr_SampleEncodeAndScoreAsPiecesr z4SentencePieceProcessor._SampleEncodeAndScoreAsPiecesc Crr&)r=SentencePieceProcessor__SampleEncodeAndScoreAsSerializedProtor=rrr&_SampleEncodeAndScoreAsSerializedProtor z=SentencePieceProcessor._SampleEncodeAndScoreAsSerializedProtoc Crr&)r to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Soothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. num_threads: number of threads in batch processing (Default = -1, auto-detected) ) model_file model_protoN) $_sentencepiece_processor_init_native _out_type_add_bos_add_eos_reverse_emit_unk_piece_enable_sampling _nbest_size_alpha _num_threadsLoad) rrTrUout_typerrrrrrrr rrrInits"zSentencePieceProcessor.Initc Cs|dur|j}|dur|j}|dur|j}|dur|j}|dur#|j}|dur*|j}|dur1|j}| dur8|j} | dur?|j} |dkrW|dusS|dksS|dksS| durWt d| dusat | t uret dt |t ur|t ur|| || ||| |||| S|tur||| ||| |||| S|dks|dkr||| ||| |||| S|d kr||| ||| |||| S|t ur||||| ||||S|tur||||| ||||S|dks|dkr||||| ||||S|d kr||||| ||||St d |) a~Encode text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str. add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: sampling parameters for unigram. Invalid in BPE-Dropout. nbest_size = {0,1}: No sampling is performed. nbest_size > 1: samples from the nbest_size results. nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. alpha: Soothing parameter for unigram sampling, and merge probability for BPE-dropout (probablity 'p' in BPE-dropout paper). num_threads: the number of threads used in the batch processing (Default = -1). NTrraWhen enable_sampling is True, We must specify "nbest_size > 1" or "nbest_size = -1", and "alpha". "nbest_size" is enabled only on unigram mode ignored in BPE-dropout. when "nbest_size = -1" , this method samples from all candidates on the lattice instead of nbest segmentations.num_threads must be intserialized_protor}immutable_protozunknown out_type={})rWrXrYrZr[r\r]r^r_ RuntimeErrorrintlistr rbr rrrrrrrW) rinputrarrrrrrrr rrrEncodesv                  zSentencePieceProcessor.EncodecK|jd|td|SNrirarrjrbrrikwargsrrrEncodeAsPieces5r-z%SentencePieceProcessor.EncodeAsPiecescKrkrlrjrgrorrr EncodeAsIds9r-z"SentencePieceProcessor.EncodeAsIdscK|jd|dd|S)NrdrmrrjrorrrEncodeAsSerializedProto=r-z.SentencePieceProcessor.EncodeAsSerializedProtocKrt)NrermrrurorrrEncodeAsImmutableProtoAr-z-SentencePieceProcessor.EncodeAsImmutableProtocK|jd|||tdd|SNTrirrrarrrnrrirrrprrrSampleEncodeAsPiecesE z+SentencePieceProcessor.SampleEncodeAsPiecescKrxryrrr{rrrSampleEncodeAsIdsJr}z(SentencePieceProcessor.SampleEncodeAsIdscK|jd|||ddd|S)NrdTrzrrur{rrrSampleEncodeAsSerializedProtoOr}z4SentencePieceProcessor.SampleEncodeAsSerializedProtocKr)NreTrzrrur{rrrSampleEncodeAsImmutableProtoTr}z3SentencePieceProcessor.SampleEncodeAsImmutableProtocsdurjdurjdurjdurjdur#jdur*jdkr0dfddt|turKfdd|DS|S)aNBestEncode text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str. add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) nbest_size: nbest size Nrrcstur|Stur|Sdks$dkr.|Sdkr<|Std)Nrdr}rezunknown out_type)rgr5rbr7r9r;rfr)rrrrrarrrr_encode}s"z3SentencePieceProcessor.NBestEncode.._encodecg|]}|qSrrrnrrrrz6SentencePieceProcessor.NBestEncode..)rWrXrYrZr[r]rrh)rrirarrrrrr)rrrrrrarrr NBestEncodeYs$ z"SentencePieceProcessor.NBestEncodecK|jd||td|SNrirrar)rrbrrirrprrrNBestEncodeAsPieces z*SentencePieceProcessor.NBestEncodeAsPiecescKrr)rrgrrrrNBestEncodeAsIdsrz'SentencePieceProcessor.NBestEncodeAsIdscK|jd||dd|S)NrdrrrrrrrNBestEncodeAsSerializedProtorz3SentencePieceProcessor.NBestEncodeAsSerializedProtocKr)NrerrrrrrrNBestEncodeAsImmutableProtorz2SentencePieceProcessor.NBestEncodeAsImmutableProtoc sdur jdur jdur jdur jdur# jdur)ddur/d dur5d dur;ddkrCtdrK sKtd f dd t|turifd d |DS|S) a SampleEncodeAndScore text input to segmented ids or tokens. Args: input: input string. accepsts list of string. out_type: output type. int or str or 'serialized_proto' or 'immutable_proto' add_bos: Add to the result (Default = false) add_eos: Add to the result (Default = false) / is added after reversing (if enabled). reverse: Reverses the tokenized sequence (Default = false) emit_unk_piece: Emits the unk literal string (Default = false) num_samples: How many samples to return (Default = 1) alpha: inverse temperature for sampling wor: whether to sample without replacement (Default = false) include_best: whether to include the best tokenization, requires wor=True (Default = false) Nrg?Frznum_examples must be positivez8When include_best is True, We must specify "wor = True".c stur|  Stur"|  Sdks*dkr7|  SdkrH|  Std)Nrdr}rezunknown output type)rgrArbrCrErGrfr) rrrrr@r>rarrr?rrrs"z._encodecrrrrrrrrrz?SentencePieceProcessor.SampleEncodeAndScore..)rWrXrYrZr[rfrrh) rrirarrrrr>rr?r@r) rrrrrr@r>rarrr?rSampleEncodeAndScores4 z+SentencePieceProcessor.SampleEncodeAndScorecK|jd|||td|SNrir>rrar)rrbrrir>rrprrrSampleEncodeAndScoreAsPieces z3SentencePieceProcessor.SampleEncodeAndScoreAsPiecescKrr)rrgrrrrSampleEncodeAndScoreAsIdsrz0SentencePieceProcessor.SampleEncodeAndScoreAsIdscK|jd|||dd|S)Nrdrrrrrrr%SampleEncodeAndScoreAsSerializedProtorz._normalizecrrrrrrrrrz4SentencePieceProcessor.Normalize..rrhrrirrrrrr Normalize z SentencePieceProcessor.NormalizecKs,i}|D] \}}t|||<q||Sr&)itemsrbrR)rrp new_kwargskeyrrrrOverrideNormalizerSpecs z-SentencePieceProcessor.OverrideNormalizerSpeccC|Sr&rrDrrr piece_sizez!SentencePieceProcessor.piece_sizecCrr&rrDrrr vocab_sizerz!SentencePieceProcessor.vocab_sizecCrr&rrDrrr __getstate__rz#SentencePieceProcessor.__getstate__cC|||dSr&rErrrrrr __setstate__z#SentencePieceProcessor.__setstate__cCrr&rrDrrrrrzSentencePieceProcessor.__len__cCs ||Sr&)rrrrrrrJz"SentencePieceProcessor.__getitem__cCs(|r|rtd|r||S||S)aOverwride SentencePieceProcessor.Load to support both model_file and model_proto. Args: model_file: The sentencepiece model file path. model_proto: The sentencepiece model serialized proto. Either `model_file` or `model_proto` must be set. z-model_file and model_proto must be exclusive.)rfrr)rrTrUrrrr`s   zSentencePieceProcessor.Load) NNNNNNNNNNN)NNNNNNr&)rd)re)ir r r0rrrrrErdelete_SentencePieceProcessorrfrrrrrrrrrrrrrrrrrrrrrrrrrr r rrrrrrrr r"r%r'r)r+r-r/r1r5r7r9r;rArCrErGrJrLrNrPrRrgrbrjrqrsrvrwr|r~rrrrrrrrrrrrrbrrrrrrrrrrrrrrrr`rrrrrs  2 ^     :    M    _        rcCrGr&)rSetRandomGeneratorSeed)seedrrrrrJrcCrGr&)rSetMinLogLevel)r<rrrrrJrc@seZdZeddddddZddZeZedd Z ed d Z ed d Z eddZ eddZ edddZedddZdS)SentencePieceTrainercCr4r&r5r6rrrr8r9zSentencePieceTrainer.cCr:r&r5r;rrrr8r=r>r?cOstd)NzNo constructor defined)r)rrrprrrrErzSentencePieceTrainer.__init__cCrGr&)r%SentencePieceTrainer__TrainFromString)rrrr_TrainFromStringrz%SentencePieceTrainer._TrainFromStringcCrGr&)r"SentencePieceTrainer__TrainFromMaprrrr _TrainFromMaprz"SentencePieceTrainer._TrainFromMapcCrnr&)r#SentencePieceTrainer__TrainFromMap2riterrrr_TrainFromMap2 z#SentencePieceTrainer._TrainFromMap2cCrGr&)r#SentencePieceTrainer__TrainFromMap3rrrr_TrainFromMap3rz#SentencePieceTrainer._TrainFromMap3cCrnr&)r#SentencePieceTrainer__TrainFromMap4rrrr_TrainFromMap4rz#SentencePieceTrainer._TrainFromMap4Nc Ks|durt|turt|Sdd}d}d}i}|D]\}}|dvr(|}q|dvr/|}q||||<q|rM|rAt||}nt|}||dS|rUt||St |S)zDTrain Sentencepiece model. Accept both kwargs and legacy string arg.NcSsZt|tur)tjddkrt}nt}tj|dd}|dd|D| St |S)zEncode value to CSV..rr)lineterminatorcSsg|]}t|qSr)rb)rr<rrrrrz@SentencePieceTrainer._Train.._encode..) rrhsysrStringIOBytesIOcsvwriterwriterowgetvaluerb)rfrrrrrs z,SentencePieceTrainer._Train.._encode)sentence_iteratorsentence_reader) model_writer) rrbrrrrrwriterr) rrprrrrrrrUrrr_Trains.      zSentencePieceTrainer._TraincKsDt|dtjdd|i|WddS1swYdS)N)ostreamrr) _LogStreamrr)r logstreamrprrrTrains "zSentencePieceTrainer.Trainr&r)r r r0rrrErr staticmethodrrrrrrrrrrrrs$      ,rc@seZdZeddddddZeZddZej Z dd Z d d Z d d Z ddZddZddZddZddZ       d"ddZd#ddZddZd d!ZdS)$SentencePieceNormalizercCr4r&r5r6rrrr8r9z SentencePieceNormalizer.cCr:r&r5r;rrrr8r=r>r?cCrAr&)r SentencePieceNormalizer_swiginitnew_SentencePieceNormalizerrDrrrrE rFz SentencePieceNormalizer.__init__cCrnr&)r/SentencePieceNormalizer_LoadFromSerializedProtorrrrr$rdz/SentencePieceNormalizer.LoadFromSerializedProtocCrnr&)r'SentencePieceNormalizer_LoadFromRuleTSV)rrrrrLoadFromRuleTSV'rdz'SentencePieceNormalizer.LoadFromRuleTSVcCrnr&)r(SentencePieceNormalizer_LoadFromRuleName)rrrrrLoadFromRuleName*rdz(SentencePieceNormalizer.LoadFromRuleNamecCrGr&)r.SentencePieceNormalizer_serialized_model_protorDrrrr-rJz.SentencePieceNormalizer.serialized_model_protocCrnr&)r$SentencePieceNormalizer_LoadFromFilerrrrr0rdz$SentencePieceNormalizer.LoadFromFilecCrnr&)r"SentencePieceNormalizer__NormalizerIrrrrJ3rdz"SentencePieceNormalizer._NormalizecCrnr&)r-SentencePieceNormalizer__NormalizeWithOffsetsrIrrrrL6rdz-SentencePieceNormalizer._NormalizeWithOffsetscCrr&)r&SentencePieceNormalizer__SetProtoFieldrrrr_SetProtoField9rz&SentencePieceNormalizer._SetProtoFieldNFc Cst||r ||}n|r||}n|r||}n |r$||}ntd|r>|d||d||d|dSdS)aInitialzie sentencePieceNormalizer. Args: model_file: The sentencepiece model file path. model_proto: The sentencepiece model serialized proto. rule_tsv: The normalization rule file in TSV format. rule_name: Pre-defined normalization name. add_dummy_prefix: add dummy prefix. escape_whitespaces: escape whitespaces. remove_extra_whitespaces: remove extra whitespaces. zno model is specifiedadd_dummy_prefixescape_whitespacesremove_extra_whitespacesN)%_sentencepiece_normalizer_init_nativerrrrrfr) rrTrUrule_tsv rule_namerrrstatusrrrrb<s      zSentencePieceNormalizer.Initcr)Ncrr&rrrrrrcrz5SentencePieceNormalizer.Normalize.._normalizecrrrrrrrrirz5SentencePieceNormalizer.Normalize..rrrrrrbrz!SentencePieceNormalizer.NormalizecCrr&rrDrrrrmrz$SentencePieceNormalizer.__getstate__cCrr&rrrrrrqrz$SentencePieceNormalizer.__setstate__)NNNNFFFr&)r r r0rrrrrErdelete_SentencePieceNormalizerrfrrrrrrJrLrrbrrrrrrrrs0 & rcCrGr&)r SetDataDir)data_dirrrrryrJr)r)rcCshi}|jD]\}}td|r"tdd|dd}|||<q|D] \}}t|||q'dS)z1Added snake_cased method from CammelCased method.z^[A-Z]+z(?._funccs*t|turfdd|DS|S)Ncsg|]}|qSrrr)rrrrrsz5_batchnize.._batched_func..rr)rrDr _batched_funcs  z!_batchnize.._batched_func)rr)rrrr)rrr _batchnizes   rrE)rrrrrrr) __version__ sentencepiece package_datac@s&eZdZdddZddZddZdS) rNcCs$||_|jdurtj|_dSdSr&)rrstderrfilenoorig_stream_fileno)rrrrrrEs z_LogStream.__init__cCs4|jdurt|j|_t|j|jdSdSr&)rosdupr orig_stream_dupdup2rrDrrr __enter__s z_LogStream.__enter__cCsD|jdur t|jt|j|jt|j|jdSdSr&)rr!closer r$r#)rrr tracebackrrr__exit__s   z_LogStream.__exit__r&)r r r0rEr%r(rrrrrs  r)>rr_swig_python_version_info __package__r rrbuiltinsr ImportErrorrr"r%r.rr/objectr3>ImmutableSentencePieceText_ImmutableSentencePiece_swigregisterri'ImmutableSentencePieceText_swigregisterr,ImmutableNBestSentencePieceText_swigregisterr#SentencePieceProcessor_swigregisterrrr!SentencePieceTrainer_swigregisterr$SentencePieceNormalizer_swigregisterrr rr!importlib.resources importlibiorrrrrErVrrrbrjTokenizer Detokenizemset_random_generator_seedset_min_log_level_versionrpathrrb resourcesfilesrrrrrsv        7 H 8 S O [