o ip@sddlZddlZddlmZmZddlZddlmZddl m Z m Z m Z ddl mZmZmZmZe r=ddlZddlmZGd d d eZGd d d e Ze ed ddGdddeZeZdS)N)Anyoverload)BasicTokenizer) ExplicitEnumadd_end_docstringsis_torch_available)ArgumentHandler ChunkPipelineDatasetbuild_pipeline_init_args),MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMESc@s&eZdZdZdeeeBfddZdS)"TokenClassificationArgumentHandlerz5 Handles arguments for token classification. inputscKs|dd}|d}|dur%t|ttfr%t|dkr%t|}t|}n$t|tr0|g}d}ntdur9t|ts?t|tjrE||d|fSt d|d}|rit|tr_t|dtr_|g}t||krit d||||fS) Nis_split_into_wordsF delimiterrr zAt least one input is required.offset_mappingz;offset_mapping should have the same batch size as the input) get isinstancelisttuplelenstrr types GeneratorType ValueError)selfrkwargsrr batch_sizerr m/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/pipelines/token_classification.py__call__s$  "      z+TokenClassificationArgumentHandler.__call__N)__name__ __module__ __qualname____doc__rrr"r r r r!rsrc@s$eZdZdZdZdZdZdZdZdS)AggregationStrategyzDAll the valid aggregation strategies for TokenClassificationPipelinenonesimplefirstaveragemaxN) r#r$r%r&NONESIMPLEFIRSTAVERAGEMAXr r r r!r'3sr'T) has_tokenizera ignore_labels (`list[str]`, defaults to `["O"]`): A list of labels to ignore. grouped_entities (`bool`, *optional*, defaults to `False`): DEPRECATED, use `aggregation_strategy` instead. Whether or not to group the tokens corresponding to the same entity together in the predictions or not. stride (`int`, *optional*): If stride is provided, the pipeline is applied on all the text. The text is split into chunks of size model_max_length. Works only with fast tokenizers and `aggregation_strategy` different from `NONE`. The value of this argument defines the number of overlapping tokens between chunks. In other words, the model will shift forward by `tokenizer.model_max_length - stride` tokens each step. aggregation_strategy (`str`, *optional*, defaults to `"none"`): The strategy to fuse (or not) tokens based on the model prediction. - "none" : Will simply not do any aggregation and simply return raw results from the model - "simple" : Will attempt to group entities following the default schema. (A, B-TAG), (B, I-TAG), (C, I-TAG), (D, B-TAG2) (E, B-TAG2) will end up being [{"word": ABC, "entity": "TAG"}, {"word": "D", "entity": "TAG2"}, {"word": "E", "entity": "TAG2"}] Notice that two consecutive B tags will end up as different entities. On word based languages, we might end up splitting words undesirably : Imagine Microsoft being tagged as [{"word": "Micro", "entity": "ENTERPRISE"}, {"word": "soft", "entity": "NAME"}]. Look for FIRST, MAX, AVERAGE for ways to mitigate that and disambiguate words (on languages that support that meaning, which is basically tokens separated by a space). These mitigations will only work on real words, "New york" might still be tagged with two different entities. - "first" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Words will simply use the tag of the first token of the word when there is ambiguity. - "average" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. scores will be averaged first across tokens, and then the maximum label is applied. - "max" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Word entity will simply be the token with the maximum score.csLeZdZdZdZdZdZdZdZe ffdd Z      d5de dBd e e eefdBd ed edBd edBf d dZededede eeeffddZede edede e eeeffddZdee eBdede eeefe e eeefBffdd Zd6ddZddZe jdfddZddZ  d7dedejd ejd e e eefdBd!ejde d"e edBdBd#e e eefdBde efd$d%Zd&e ede de efd'd(Zd)e ede defd*d+Zd)e ede de efd,d-Z d)e edefd.d/Z!d0ede eeffd1d2Z"d)e ede efd3d4Z#Z$S)8TokenClassificationPipelineuv Named Entity Recognition pipeline using any `ModelForTokenClassification`. See the [named entity recognition examples](../task_summary#named-entity-recognition) for more information. Example: ```python >>> from transformers import pipeline >>> token_classifier = pipeline(model="Jean-Baptiste/camembert-ner", aggregation_strategy="simple") >>> sentence = "Je m'appelle jean-baptiste et je vis à montréal" >>> tokens = token_classifier(sentence) >>> tokens [{'entity_group': 'PER', 'score': 0.9931, 'word': 'jean-baptiste', 'start': 12, 'end': 26}, {'entity_group': 'LOC', 'score': 0.998, 'word': 'montréal', 'start': 38, 'end': 47}] >>> token = tokens[0] >>> # Start and end provide an easy way to highlight words in the original text. >>> sentence[token["start"] : token["end"]] ' jean-baptiste' >>> # Some models use the same idea to do part of speech. >>> syntaxer = pipeline(model="vblagoje/bert-english-uncased-finetuned-pos", aggregation_strategy="simple") >>> syntaxer("My name is Sarah and I live in London") [{'entity_group': 'PRON', 'score': 0.999, 'word': 'my', 'start': 0, 'end': 2}, {'entity_group': 'NOUN', 'score': 0.997, 'word': 'name', 'start': 3, 'end': 7}, {'entity_group': 'AUX', 'score': 0.994, 'word': 'is', 'start': 8, 'end': 10}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'sarah', 'start': 11, 'end': 16}, {'entity_group': 'CCONJ', 'score': 0.999, 'word': 'and', 'start': 17, 'end': 20}, {'entity_group': 'PRON', 'score': 0.999, 'word': 'i', 'start': 21, 'end': 22}, {'entity_group': 'VERB', 'score': 0.998, 'word': 'live', 'start': 23, 'end': 27}, {'entity_group': 'ADP', 'score': 0.999, 'word': 'in', 'start': 28, 'end': 30}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'london', 'start': 31, 'end': 37}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This token recognition pipeline can currently be loaded from [`pipeline`] using the following task identifier: `"ner"` (for predicting the classes of tokens in a sequence: person, organisation, location or miscellaneous). The models that this pipeline can use are models that have been fine-tuned on a token classification task. See the up-to-date list of available models on [huggingface.co/models](https://huggingface.co/models?filter=token-classification). sequencesFTc s2tjdi||ttdd|_||_dS)NF) do_lower_caser )super__init__check_model_typerr_basic_tokenizer _args_parser)r args_parserr __class__r r!r7s   z$TokenClassificationPipeline.__init__Naggregation_strategyrrstriderc Csi}||d<|r|durdn||d<|dur||d<i}|durAt|tr+t|}|tjtjtjhvr=|jjs=t d||d<|durI||d<|durw||jj krWt d|tj krdt d |d |jjrsd d |d } | |d <nt d|i|fS)Nr rrz{Slow tokenizers cannot handle subwords. Please set the `aggregation_strategy` option to `"simple"` or use a fast tokenizer.r> ignore_labelszl`stride` must be less than `tokenizer.model_max_length` (or even lower if the tokenizer adds special tokens)zI`stride` was provided to process all the text but `aggregation_strategy="z&"`, please select another one instead.T)return_overflowing_tokenspaddingr?tokenizer_paramszm`stride` was provided to process all the text but you're using a slow tokenizer. Please use a fast tokenizer.) rrr'upperr/r1r0 tokenizeris_fastrmodel_max_lengthr-) rrAr>rrr?rpreprocess_paramspostprocess_paramsrDr r r!_sanitize_parameterssT       z0TokenClassificationPipeline._sanitize_parametersrrreturncKdSNr rrrr r r!r"z$TokenClassificationPipeline.__call__cKrMrNr rOr r r!r"rPc sv|j|fi|\}}}}||d<||d<|r+tdd|Ds+tj|gfi|S|r1||d<tj|fi|S)a Classify each token of the text(s) given as inputs. Args: inputs (`str` or `List[str]`): One or several texts (or one list of texts) for token classification. Can be pre-tokenized when `is_split_into_words=True`. Return: A list or a list of list of `dict`: Each result comes as a list of dictionaries (one for each token in the corresponding input, or each entity if this pipeline was instantiated with an aggregation_strategy) with the following keys: - **word** (`str`) -- The token/word classified. This is obtained by decoding the selected tokens. If you want to have the exact string in the original sentence, use `start` and `end`. - **score** (`float`) -- The corresponding probability for `entity`. - **entity** (`str`) -- The entity predicted for that token/word (it is named *entity_group* when *aggregation_strategy* is not `"none"`. - **index** (`int`, only present when `aggregation_strategy="none"`) -- The index of the corresponding token in the sentence. - **start** (`int`, *optional*) -- The index of the start of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer - **end** (`int`, *optional*) -- The index of the end of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer rrcss|]}t|tVqdSrN)rr).0inputr r r! sz7TokenClassificationPipeline.__call__..r)r:allr6r")rrr_inputsrrrr<r r!r"sc+s|di}|jjo|jjdk}d}|d}|rT|d}t|ts&td|} || }g}t|} d} | D]} || | t| f| t| | 7} q7| } d|d<n t|t s]td|} |j| fd|d|jj d |}|ry|jj sytd |d dt|d }t |D]9fd d| D}|dur||d<dkr|nd|d<|dk|d<|dur| |d<||d<|VqdS)NrDrrrzEWhen `is_split_into_words=True`, `sentence` must be a list of tokens.TzKWhen `is_split_into_words=False`, `sentence` must be an untokenized string.pt)return_tensors truncationreturn_special_tokens_maskreturn_offsets_mappingz@is_split_into_words=True is only supported with fast tokenizers.overflow_to_sample_mapping input_idscs i|] \}}||dqS)r) unsqueeze)rQkvir r! %s z:TokenClassificationPipeline.preprocess..rsentencer is_lastword_idsword_to_chars_map)poprFrHrrrjoinrappendrrGrangeitemsre)rrcrrIrDrXrfrrwords delimiter_len char_offsetwordtext_to_tokenizer num_chunks model_inputsr r`r! preprocesss^         z&TokenClassificationPipeline.preprocessc Cs|d}|dd}|d}|d}|dd}|dd}|jd i|}t|tr2|dn|d} | ||||||d |S) Nspecial_tokens_maskrrcrdrerflogitsr)rurtrrcrdrerfr )rgmodelrdict) rrrrtrrcrdrerfoutputrur r r!_forward1s$      z$TokenClassificationPipeline._forwardc sLdurdgg}|dd}|D]}|ddjtjtjfvr.|ddtj}n|dd}|dd}|dd} |ddurN|ddnd} |dd} |d } tj |d d d } t || }||j d d d }|j || || | || |d }| ||}fdd|D}||qt|}|dkr||}|S)NOrrfrurcr\rrtreT)axiskeepdims)rerfcs0g|]}|ddvr|ddvr|qS)entityN entity_group)rrQr~rAr r! ns z;TokenClassificationPipeline.postprocess..r )rdtypetorchbfloat16float16tofloat32numpynpr,expsumgather_pre_entities aggregateextendraggregate_overlapping_entities)r all_outputsr>rA all_entitiesrf model_outputsrurcr\rrtremaxes shifted_expscores pre_entitiesgrouped_entitiesentitiesrqr rr! postprocessHsH      z'TokenClassificationPipeline.postprocesscCst|dkr|St|ddd}g}|d}|D]>}|d|dkr*|dkrOnn#|d|d}|d|d}||ksL||krN|d|dkrN|}q|||}q|||S)NrcSs|dS)Nstartr )xr r r!}szLTokenClassificationPipeline.aggregate_overlapping_entities..keyrendscore)rsortedri)rraggregated_entitiesprevious_entityr~current_lengthprevious_lengthr r r!rzs$ $  z:TokenClassificationPipeline.aggregate_overlapping_entitiesrcr\rrtrerfc Csfg} t|D]\} } || rq|jt|| } |dur|| \} }|durA|durA|| }|durA||\}}| |7} ||7}t| tsN| } |}|| |}t|jddrmt|jjjddrmt | t |k}n |t j t j t j hvr}tdt| dkod|| d| dv}t|| |jjkr|} d}nd} d}d}| | | || |d }| |q| S) zTFuse various numpy arrays into dicts with all the information needed for aggregationN _tokenizercontinuing_subword_prefixz?Tokenizer does not support real words, using fallback heuristicrr@r F)rorrrindex is_subword) enumeraterFconvert_ids_to_tokensintritemgetattrrrvrr'r/r0r1warningswarn UserWarning unk_token_idri)rrcr\rrrtr>rerfridx token_scoresro start_indend_ind word_index start_char_word_refr pre_entityr r r!rs^        z/TokenClassificationPipeline.gather_pre_entitiesrcCs|tjtjhvr7g}|D])}|d}|d|}|jjj|||d|d|d|dd}||q n|||}|tjkrD|S| |S)Nrrrorr)r~rrrorr) r'r-r.argmaxrvconfigid2labelriaggregate_wordsgroup_entities)rrr>rr entity_idxrr~r r r!rs$      z%TokenClassificationPipeline.aggregaterc Cs|jdd|D}|tjkr&|dd}|}||}|jjj|}nK|tjkrGt |ddd}|d}|}||}|jjj|}n*|tj krmt dd|D}t j |dd } | } |jjj| }| | }ntd ||||dd |d d d} | S)NcSg|]}|dqSror rr r r!rz>TokenClassificationPipeline.aggregate_word..rrcSs |dS)Nr)r,)r~r r r!rs z.rcSr)rr rr r r!rr)r|zInvalid aggregation_strategyrr{r)r~rrorr)rFconvert_tokens_to_stringr'r/rrvrrr1r,r0rstacknanmeanr) rrr>rorrrr~ max_entityaverage_scoresr new_entityr r r!aggregate_words4       z*TokenClassificationPipeline.aggregate_wordcCs|tjtjhvr tdg}d}|D] }|dur|g}q|dr&||q|||||g}q|dur@|||||S)z Override tokens from a given word that disagree to force agreement on word boundaries. Example: micro|soft| com|pany| B-ENT I-NAME I-ENT I-ENT will be rewritten with first strategy as microsoft| company| B-ENT I-ENT z;NONE and SIMPLE strategies are invalid for word aggregationNr)r'r-r.rrir)rrr> word_entities word_groupr~r r r!r s" z+TokenClassificationPipeline.aggregate_wordscCsl|ddddd}tdd|D}dd|D}|t||j||dd |dd d }|S) z Group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. rr~-r r{cSr)rr rr r r!r1rzBTokenClassificationPipeline.group_sub_entities..cSrrr rr r r!r2rrr)rrrorr)splitrrmeanrFr)rrr~rtokensrr r r!group_sub_entities(s   z.TokenClassificationPipeline.group_sub_entities entity_namecCsT|drd}|dd}||fS|dr"d}|dd}||fSd}|}||fS)NzB-BrzI-I) startswith)rrbitagr r r!get_tag=s    z#TokenClassificationPipeline.get_tagc Csg}g}|D]7}|s||q||d\}}||dd\}}||kr2|dkr2||q||||g}q|rH||||S)z Find and group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. r~r{r)rirr) rr entity_groupsentity_group_disaggr~rrlast_bilast_tagr r r!rKs  z*TokenClassificationPipeline.group_entities)NNNFNNrN)NN)%r#r$r%r&default_input_names_load_processor_load_image_processor_load_feature_extractor_load_tokenizerrr7r'rrrboolrrKrrrwr"rsryr-rrrndarrayrrrrrrr __classcell__r r r<r!r3=sz#$  ;$,B %82  H"r3)rrtypingrrrr$models.bert.tokenization_bert_legacyrutilsrrrbaser r r r rmodels.auto.modeling_autorrr'r3 NerPipeliner r r r!s,   "