o i@sddlZddlZddlZddlmZmZddlmZddlZddl m Z ddl m Z ddl mZddlmZmZd d lmZmZmZd d lmZeeZeGd d d ZGdddeZGddde ZdS)N) dataclassfield)Enum)FileLock)Dataset)PreTrainedTokenizerBase)check_torch_load_is_safelogging)!glue_convert_examples_to_featuresglue_output_modesglue_processors) InputFeaturesc@seZdZUdZedddeidZe e d<eddidZ e e d<ed dd id Z e e d <ed ddid Zee d<ddZdS)GlueDataTrainingArgumentsz Arguments pertaining to what data we are going to input our model for training and eval. Using `HfArgumentParser` we can turn this class into argparse arguments to be able to specify them on the command line. helpz"The name of the task to train on: z, )metadata task_namezUThe input data dir. Should contain the .tsv files (or other data files) for the task.data_dirzThe maximum total input sequence length after tokenization. Sequences longer than this will be truncated, sequences shorter will be padded.)defaultrmax_seq_lengthFz1Overwrite the cached training and evaluation setsoverwrite_cachecCs|j|_dSN)rlowerselfra/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/data/datasets/glue.py __post_init__<sz'GlueDataTrainingArguments.__post_init__N)__name__ __module__ __qualname____doc__rjoinrkeysrstr__annotations__rrintrboolrrrrrr"s $  rc@seZdZdZdZdZdS)SplittraindevtestN)r r!r"r+r,r-rrrrr*@sr*c @s|eZdZUeed<eed<eeed<dej dfdede de dBdeeBdedBf d d Z d d Z d efddZddZdS) GlueDatasetargs output_modefeaturesN tokenizer limit_lengthmode cache_dirc Cs&tdt||_t|j|_t|j|_t |t r-zt |}Wn t y,t dwt j|dur6|n|jd|jd|jjd|jd|j}|j}|jdvri|jjdvri|d|d|d<|d<||_|d }t|t j|r|jst} ttj|d d |_t d |d t| ndt d|j|t j!kr|j"|j} n|t j#kr|j$|j} n|j%|j} |dur| d|} t&| ||j||jd|_t} t'|j|t d|dt| ddWddSWddS1s wYdS)NaThis dataset will be removed from the library soon, preprocessing should be handled with the Hugging Face Datasets library. You can have a look at this example script for pointers: https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.pyzmode is not a valid split namecached__)mnlizmnli-mm)RobertaTokenizerXLMRobertaTokenizer BartTokenizerBartTokenizerFastr z.lockT) weights_onlyz"Loading features from cached file z [took %.3f s]z'Creating features from dataset file at ) max_length label_listr0z!Saving features into cached file z [took z.3fz s])(warningswarn FutureWarningr/rr processorr r0 isinstancer&r*KeyErrorospathr$rvalue __class__r r get_labelsr@rexistsrtimer torchloadr1loggerinfor,get_dev_examplesr-get_test_examplesget_train_examplesr save) rr/r2r3r4r5cached_features_filer@ lock_pathstartexamplesrrr__init__Ksj    $     $zGlueDataset.__init__cCs t|jSr)lenr1rrrr__len__ zGlueDataset.__len__returncCs |j|Sr)r1)rirrr __getitem__r]zGlueDataset.__getitem__cCs|jSr)r@rrrrrKszGlueDataset.get_labels)r r!r"rr'r&listrr*r+rr(rZr\r`rKrrrrr.Fs*   J r.)rGrMrA dataclassesrrenumrrNfilelockrtorch.utils.datartokenization_utils_baserutilsr r processors.gluer r rprocessors.utilsr get_loggerr rPrr*r.rrrrs"