o i@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZddlmZmZddlmZdd lmZmZmZmZmZdd lm Z m!Z!e"e#Z$d e%d e%fd dZ&e 'Z(ddZ)d e%ej*Bd dfddZ+de%ej*Bd e,e%fddZ-de%ej*Bd e,e%fddZ.de%ej*Bd e,e%fddZ/de%ej*Bd e,e%fddZ0ddde%d e%ej*Bd!e1d e2fd"d#Z3        d@d$e%ej*Bde%d%e%ej*BdBd&e1d'e4e%e%fdBd(e1e%BdBd)e%dBd*e1d+e%dBd,e%dBd e%fd-d.Z5        d@d/e%d$e%ej*Bd%e%ej*BdBd&e1d'e4e%e%fdBd(e1e%BdBd)e%dBd*e1d+e%dBd0e%dBd e2fd1d2Z6dAd3ed4e%ej*Bd5e4dBd e,e%fd6d7Z7d8d9Z8d:Z9 dBd;d<Z:dCd>d?Z;dS)Dz3Utilities to dynamically load objects from the Hub.N)Path) ModuleType)Any)is_offline_modetry_to_load_from_cache)version)HF_MODULES_CACHE TRANSFORMERS_DYNAMIC_MODULE_NAME cached_fileextract_commit_hashlogging)VersionComparisonsplit_package_versionnamereturncCsx|dddd}|r|drd|}t|r*td|d|d |S|s:td|d|d |S) a Tries to sanitize a module name so that it can be used as a Python module. The following transformations are applied: 1. Replace `.` in module names with `_dot_`. 2. Replace `-` in module names with `_hyphen_`. 3. If the module name starts with a digit, prepend it with `_`. 4. Warn if the sanitized name is a Python reserved keyword or not a valid identifier. If the input name is already a valid identifier, it is returned unchanged. ._dot_-_hyphen_r_zThe module name z (originally z\) is a reserved keyword in Python. Please rename the original module to avoid import issues.z]) is not a valid Python identifier. Please rename the original module to avoid import issues.)replaceisdigitkeyword iskeywordloggerwarning isidentifier)rnew_namerc/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/dynamic_module_utils.py_sanitize_module_name1s   r!cCsTttjvrdStjttjtddttd}|s(|t dSdS)z_ Creates the cache directory for modules with an init, and adds it to the Python path. NTexist_ok __init__.py) r syspathappendosmakedirsrexiststouch importlibinvalidate_caches) init_pathrrr init_hf_modulesUs    r/cCs`ttt|}|jst|jtj|dd|d}|s.| t dSdS)z Creates a dynamic module in the cache directory for modules. Args: name (`str` or `os.PathLike`): The name of the dynamic module to create. Tr"r$N) r/rr resolveparentr*create_dynamic_moduler(r)r+r,r-)rdynamic_module_pathr.rrr r2es   r2 module_filecCsht|dd }|}Wdn1swYtjd|tjd}|tjd|tjd7}tt|S)z Get the list of modules that are relatively imported in a module file. Args: module_file (`str` or `os.PathLike`): The module file to inspect. Returns: `list[str]`: The list of relative imports in the module. utf-8encodingNz^\s*import\s+\.(\S+)\s*$)flagsz^\s*from\s+\.(\S+)\s+import)openreadrefindall MULTILINElistset)r4fcontentrelative_importsrrr get_relative_imports{s   rCcszd}|g}g|s;g}|D] }|t|q t|jfdd|D}fdd|D}t|dk}||r S)a Get the list of all files that are needed for a given module. Note that this function recurses through the relative imports (if a imports b and b imports c, it will return module files for b and c). Args: module_file (`str` or `os.PathLike`): The module file to inspect. Returns: `list[str]`: The list of all relative imports a given module needs (recursively), which will give us the list of module files a given module needs. Fcsg|] }t|dqS).py)str).0m) module_pathrr sz-get_relative_import_files..csg|]}|vr|qSrrrFr@)all_relative_importsrr rIsr)extendrCrr1len)r4 no_changefiles_to_check new_importsr@new_import_filesr)rKrHr get_relative_import_filess     rRfilenamecslt|dd }|}Wdn1swYtddlfddt|}|tS)a Extracts all the libraries (not relative imports this time) that are imported in a file. Args: filename (`str` or `os.PathLike`): The module file to inspect. Returns: `list[str]`: The list of all packages required to use the input module. r5r6Nrcst|tjrdSt|tjr;|j}t|D]#}t|tjr9t|jdd}| dr/| ds6t j j |r9dSqn9t|tjrW|jD]}|jdd}|rU|qDnt|tjrt|jdkrt|jrt|jdd}|rt|t|D]}|qydS)Nid available is_flash_attnrr) isinstanceastTryIftestwalkCallgetattrfuncendswith startswithhasattrutils import_utilsImportnamesrsplitadd ImportFromlevelmoduleiter_child_nodes)noder\condition_nodecheck_functionalias top_modulechildimported_modulesrecursive_look_for_imports transformersrr rvs>         z/get_imports..recursive_look_for_imports)r9r:r?transformers.utilsrYparsesorted)rSr@rAtreerrtr get_importss  !r|c Cst|}g}|D]3}zt|Wqty;}ztd|d|dt|vr0||nWYd}~qd}~wwt|dkrStdd |dd |d t |S) a. Check if the current Python environment contains all the libraries that are imported in a file. Will raise if a library is missing. Args: filename (`str` or `os.PathLike`): The module file to check. Returns: `list[str]`: The list of relative imports in the file. z&Encountered exception while importing z: zNo module namedNrz\This modeling file requires the following packages that were not found in your environment: z, z. Run `pip install  `) r|r, import_module ImportErrorrrrEr'rMjoinrC)rSimportsmissing_packagesimp exceptionrrr check_importss,     rF force_reload class_namerHrc Cstj|}|d}|tjjd}tt|}tf|r)t j |dt t j |}t jj||d}|gtttt|}tddd|D}|durbt j|} | t j |<n|} t| dd |kru|j| || _t| |WdS1swYdS) a Import a module on the cache directory for modules and extract a class from it. Args: class_name (`str`): The name of the class to import. module_path (`str` or `os.PathLike`): The path to the module to import. force_reload (`bool`, *optional*, defaults to `False`): Whether to reload the dynamic module from file if it already exists in `sys.modules`. Otherwise, the module is only reloaded if the file has changed. Returns: `typing.Type`: The class looked for. rDrN)locationcss |] }t||VqdSN)bytes read_bytesrJrrr *sz&get_class_in_module..__transformers_module_hash__rU)r(r&normpath removesuffixrseprr _HF_REMOTE_CODE_LOCKr%modulespopr,r-getutilspec_from_file_locationrzmaprRhashlibsha256r hexdigestmodule_from_specr_loader exec_moduler) rrHrrr4 cached_module module_spec module_files module_hashrlrrr get_class_in_module s*       $rpretrained_model_name_or_path cache_dirforce_downloadproxiestokenrevisionlocal_files_only repo_type _commit_hashc  Kstr |s tdd}t|}tj|} | r!ttj|} ntjj t t| d} t |||| |d} g}zt|||||||||| d }| sS| |krS||Wntyhtd|d|dwt|}ttjj | }t|tt|}| ttj|kr||rt|t||s||jjddd t|||t|D]/}t|j|d }tj ||}||rt|t||st|||tqnbt|| }||}|tjj |}tj ||}tt|j||st|||t|D]-}||j|d sBt |t|j|d |||||||d ||d qt!|d kry|d uryd dd|D}|d uradn|d}d||}t"d|d|dtj ||S)a Prepares Downloads a module from a local folder or a distant repo and returns its path inside the cached Transformers module. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. module_file (`str`): The name of the module file containing the class to look for. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. repo_type (`str`, *optional*): Specify the repo type (useful when downloading from a space for instance). Passing `token=True` is required when you want to use a private model. Returns: `str`: The path to the module inside the cache. z+Offline mode: forcing local_files_only=TrueT/)rrr)rrrrrrrrzCould not locate the z inside r)parentsr#rD)rrrrrrrrN cSsg|]}d|qS)z- rrJrrr rIsz*get_cached_module_file..rUzs/zhttps://huggingface.co/z9A new version of the following files was downloaded from z: z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.)#rrinforEr(r&isdirr!basenamerrrrhrr r'OSErrorrr r2rr r*filecmpcmpr1mkdirshutilcopyr,r-r get_cached_module_filerMr)rr4rrrrrrrrdeprecated_kwargsis_local submoduler new_filesresolved_module_filemodules_neededfull_submodulesubmodule_path module_neededmodule_needed_file commit_hashfull_submodule_module_file_path repo_type_strurlrrr r:s :          rclass_reference code_revisionc  Ksjd|vr |d\} }n|} |d\} } | dur|| kr|} t| | d||||| ||d }t| ||dS)a Extracts a class from a module file, present in the local folder or repository of a model. Calling this function will execute the code in the module file found locally or downloaded from the Hub. It should therefore only be called on trusted repos. Args: class_reference (`str`): The full name of the class to load, including its module and optionally its repo. pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. This is used when `class_reference` does not specify another repo. module_file (`str`): The name of the module file containing the class to look for. class_name (`str`): The name of the class to import in the module. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. repo_type (`str`, *optional*): Specify the repo type (useful when downloading from a space for instance). code_revision (`str`, *optional*, defaults to `"main"`): The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. Passing `token=True` is required when you want to use a private model. Returns: `typing.Type`: The class, dynamically imported from the module. Examples: ```python # Download module `modeling.py` from huggingface.co and cache then extract the class `MyBertModel` from this # module. cls = get_class_from_dynamic_module("modeling.MyBertModel", "sgugger/my-bert-model") # Download module `modeling.py` from a given repo and cache then extract the class `MyBertModel` from this # module. cls = get_class_from_dynamic_module("sgugger/my-bert-model--modeling.MyBertModel", "sgugger/another-bert-model") ```z--rNrD)rrrrrrrr)rhrr)rrrrrrrrrrkwargsrepo_idr4r final_modulerrr get_class_from_dynamic_modules$W robjfolderconfigc sjdkrtdd|ddSfdd}t|ttfr*|D]}||q"n|dur2||g}tjjj}t |t |j }t ||| |t|D]}t |t |j }t ||| |qS|S)a  Save the modeling files corresponding to a custom model/configuration/tokenizer etc. in a given folder. Optionally adds the proper fields in a config. Args: obj (`Any`): The object for which to save the module files. folder (`str` or `os.PathLike`): The folder where to save. config (`PreTrainedConfig` or dictionary, `optional`): A config in which to register the auto_map corresponding to this custom object. Returns: `list[str]`: The list of files saved. __main__z We can't save the code defining z in z as it's been defined in __main__. You should put this code in a separate module so we can include it in the saved folder and make it easier to share via the Hub.Nc sjj}|dd}|djj}d|vrYd}d}jjdrL|djj}tdddurKtd}|j}|dd}|d|j}n |djj}||f}t|tro|di} || j <| |d<dSt|dddur||j j <dSj |i|_ dS)Nr TokenizerFastslow_tokenizer_classauto_map) __class__ __module__rh__name__rar_rXdictr _auto_classr) _config module_name last_module full_namerfast_tokenizer_classslow_tokenizerslow_tok_module_namelast_slow_tok_modulerrrr _set_auto_map_in_config`s.     z3custom_object_save.._set_auto_map_in_config)rrrrXr>tupler%r__file__rrrrr'rR) rrrrcfgresult object_file dest_file needed_filerrr custom_object_saveJs,         rcCstd)NzLoading this model requires you to execute custom code contained in the model repository on your local machine. Please set the option `trust_remote_code=True` to permit loading of this model.) ValueError)signumframerrr _raise_timeout_errorsrc Cs|dur2|durd|d|d|d}ntj|r)d|dtj|d}n d|d|d}|dur|r;d}n||rtd krd}zZz6ttjt}tt|durtt |d |d }| d vrhd }n| dvrpd}|dusUtd Wnt yt |d |dwW|durttj|td n|durttj|td ww|rtdd|r|s|st |d |d|S)a Resolves the `trust_remote_code` argument. If there is remote code to be loaded, the user must opt-in to loading it. Args: trust_remote_code (`bool` or `None`): User-defined `trust_remote_code` value. model_name (`str`): The name of the model repository in huggingface.co. has_local_code (`bool`): Whether the model has local code. has_remote_code (`bool`): Whether the model has remote code. error_message (`str`, *optional*): Custom error message to display if there is remote code to load and the user didn't opt-in. If unset, the error message will be regarding loading a model with custom code. Returns: The resolved `trust_remote_code` value. NzThe repository z% references custom code contained in zm which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/z . zt contains custom code which must be executed to correctly load the model. You can inspect the repository content at z contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/Frz9 You can inspect the repository content at https://hf.co/z. You can avoid this prompt in future by passing the argument `trust_remote_code=True`. Do you wish to run the custom code? [y/N] )yesy1T)non0rUzS. Please pass the argument `trust_remote_code=True` to allow custom code to be run.) r(r&rabspathTIME_OUT_REMOTE_CODEsignalSIGALRMralarminputlower Exceptionr)trust_remote_code model_namehas_local_codehas_remote_code error_message upstream_repoprev_sig_handleranswerrrr resolve_trust_remote_codesl           r requirements.txtc Ks\g}ztd ||d|}t|d }|}Wdn1s!wY|D]i}|}|r5|dr6q(z t|\}}} WntyN|}d\}} Ynwztj |} Wntjj yj| |dYq(w|dur| durt |t | t | } nd} | s| |d| d q(Wn tyYnw|rtd |d d |dS)a Tries to locate `requirements_file` in a local folder or repo, and confirms that the environment has all the python dependencies installed. Args: path_or_repo_id (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a model repo on huggingface.co. - a path to a *directory* potentially containing the file. kwargs (`dict[str, Any]`, *optional*): Additional arguments to pass to `cached_file`. )path_or_repo_idrSrN#NNz (installed: None)Tz (installed: )z4Missing requirements in your local environment for `z`: rr)r r9 readlinesstriprbrrr,metadatarPackageNotFoundErrorr'r from_stringryrrr) r requirements_filerfailed requirementsr@ requirement package_name delimiterversion_numberlocal_package_version is_satisfiedrrr check_python_requirementssN      r )NFNNNFNNrr)r )<__doc__rYrrr,importlib.metadataimportlib.utilrr(r;rrr% threadingpathlibrtypesrtypingrhuggingface_hubrr packagingrrdr r r r r utils.import_utilsrr get_loggerrrrEr!Lockrr/PathLiker2r>rCrRr|rbooltyperrrrrrrr r rrrr s      ! 7' 3     &     *nL U