o 0iaO@sddlZddlZddlZddlZddlmZddlmZddlmZm Z ddl m Z ddl m Z ddlZddlZddlmZddlmZd d lmZd d lmZd d lmZd dlmZd dlmZd dl m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'd dl(m)Z)d dl*m+Z+ee,Z-e$rddl.m/m0Z1dej2j3de4fddZ5dej2j3de4fddZ6dej2j3de4fddZ7dej2j3de4fddZ8dej2j3dej2j3fddZ9dej2j3fd d!Z:d"ej2j3de4fd#d$Z; &dLd'e4d(e4d)e4fd*d+Zfd/d0Z?dMd1e4d2e4fd3d4Z@e"d5r ejAnej/ZBeBjCjDejEeejFgZGe"d6r"eGHejIjJdNd7d8ZKd9d:ZLd;d<ZMdNd=e eNde4fd>d?ZOdeNfd@dAZPdBdCZQdDdEZRdFeSfdGdHZTdOd"ej2j3dIe4deUej2j3fdJdKZVdS)PN)encode) OrderedDict)partialreduce) MethodType)Optional)Version) save_file)write_basic_config) get_logger) PartialState)FSDP_PYTORCH_VERSION)DistributedType)is_deepspeed_availableis_numpy_availableis_torch_distributed_availableis_torch_xla_availableis_weights_only_available)id_tensor_storage) convert_model)is_torch_versionmodulereturncCsttdsdSt|tjjjS)zD Check whether the module was compiled with torch.compile() _dynamoF)hasattrtorch isinstancer eval_frameOptimizedModulerr"X/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/accelerate/utils/other.pyis_compiled_module6s r$cCs<ttdsdS|jr|D] }t|tjjjrdSqdS)z\ Check whether the module has submodules that were compiled with `torch.compile()`. rFT)rr_modulesmodulesrrrr r submoduler"r"r#has_compiled_regions@s  r)cs$ttjjotfddDS)z Check whether the module is a repeated block, i.e. `torch.nn.ModuleList` with all children of the same class. This is useful to determine whether we should apply regional compilation to the module. c3s |] }t|djVqdS)rN)r __class__).0mr!r"r# Usz%is_repeated_blocks..)rrnn ModuleListallr!r"r!r#is_repeated_blocksOs$r1cCs&|jr|D] }t|rdSqdS)z Check whether the module has repeated blocks, i.e. `torch.nn.ModuleList` with all children of the same class, at any level of the module hierarchy. This is useful to determine whether we should apply regional compilation to the module. TF)r%r&r1r'r"r"r#has_repeated_blocksXs  r2c sFdtjjdtjjffdd |fi|}d|jvr!||jd<|S)a_ Performs regional compilation where we target repeated blocks of the same class and compile them sequentially to hit the compiler's cache. For example, in `GPT2LMHeadModel`, the repeated block/class is `GPT2Block`, and can be accessed as `model.transformer.h[0]`. The rest of the model (e.g. model.lm_head) is compiled separately. This allows us to speed up the compilation overhead / cold start of models like LLMs and Transformers in general. See https://pytorch.org/tutorials/recipes/regional_compilation.html for more details. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `torch.compile()`. Returns: `torch.nn.Module`: A new instance of the model with some compiled regions. Example: ```python >>> from accelerate.utils import compile_regions >>> from transformers import AutoModelForCausalLM >>> model = AutoModelForCausalLM.from_pretrained("gpt2") >>> compiled_model = compile_regions(model, mode="reduce-overhead") >>> compiled_model.transformer.h[0] OptimizedModule( (_orig_mod): GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D(nf=2304, nx=768) (c_proj): Conv1D(nf=768, nx=768) (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D(nf=3072, nx=768) (c_proj): Conv1D(nf=768, nx=3072) (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) ``` rrcst|rtj}|D]}|tj|fi|q |St|rH|j|j}|j |j i|_ | D]\}}| ||fi|q5|Stj|fi|}|SN)r1rr.r/appendcompiler2r*__new____dict__updater%named_children add_module)rcompile_kwargs new_moduler(name_compile_regionsr"r#r?s  z)compile_regions.._compile_regions _orig_mod)rr.Moduler7)rr;r<r"r>r#compile_regionsfs /  rBcKsdt|r|D] }|jdi|qdSt|r(|D] }t|fi|qdS|jdi|dS)a Performs regional compilation the same way as `compile_regions`, but specifically for `DeepSpeedEngine.module`. Since the model is wrapped in a `DeepSpeedEngine` and has many added hooks, offloaded parameters, etc that `torch.compile(...)` interferes with, version of trgional compilation uses the inplace `module.compile()` method instead. Args: module (`torch.nn.Module`): The model to compile. **compile_kwargs: Additional keyword arguments to pass to `module.compile()`. Nr")r1r5r2childrencompile_regions_deepspeed)rr;r(childr"r"r#rDs rDmodelcs>tddr ddlmnddlmtfdd|DS)z Check if the model has DTensor parameters. Args: model (`torch.nn.Module`): The model to check. Returns: `bool`: Whether the model has DTensor parameters. >=z2.5.0rDTensorc3s|]}t|VqdSr3)r)r+prHr"r#r-sz$model_has_dtensor..)rtorch.distributed.tensorrItorch.distributed._tensorany parameters)rFr"rHr#model_has_dtensors  rOTFkeep_fp32_wrapperkeep_torch_compile recursivec sLtjjjtjjf}t|}t|}|r|}|j}n |r"|}|jd}t r0ddl m }||f7}t dt rCtrCddlm} || f7}t||rP|j}t||sH|r\fdd|}|s|j} |jdd } | d urt| d r~| j} | | krynt| d sqt| ||_t|d d rt|d d |r|r||_|}|S|r||jd<|}|S)a Extract a model from its distributed containers. Args: model (`torch.nn.Module`): The model to extract. keep_fp32_wrapper (`bool`, *optional*): Whether to remove mixed precision hooks from the model. keep_torch_compile (`bool`, *optional*): Whether to unwrap compiled model. recursive (`bool`, *optional*, defaults to `False`): Whether to recursively extract all cases of `module.module` from `model` as well as unwrap child sublayers recursively, not just the top-level distributed containers. Returns: `torch.nn.Module`: The extracted model. r@r)DeepSpeedEnginerG)FullyShardedDataParallelcs@t|dr |j}n|}|D] \}}t|||q|S)Nr)rrr9setattr)runwrapped_moduler=rE_recursive_unwrapr"r#rX s  z6extract_model_from_parallel.._recursive_unwrap_original_forwardN __wrapped__ _converted_to_transformer_engineF)to_transformer_engine)rr.parallelDistributedDataParallel DataParallelr$r)r@r7r deepspeedrSrrr2torch.distributed.fsdp.fully_sharded_data_parallelrTrrforwardpoprrZrgetattrr) rFrPrQrRoptions is_compiled has_compiledcompiled_modelrSFSDPrboriginal_forwardr"rWr#extract_model_from_parallelsT              rkcCstdS)a  Introduces a blocking point in the script, making sure all processes have reached this point before continuing. Make sure all processes will reach this instruction otherwise one of your processes will hang forever. N)r wait_for_everyoner"r"r"r#rl/s rl state_dictcstt}D]\}}t|ts|t||q dd|D}t}| D] }fdd|D}| |dd|ddD]}|=qFq,t |dkr\t d|d d dDS) z Cleans the state dictionary from a model and removes tensor aliasing if present. Args: state_dict (`dict`): The state dictionary from a model cSs"i|] \}}t|dkr||qS)r)len)r+ptrnamesr"r"r# Ks"z4clean_state_dict_for_safetensors..csg|]}|vr|qSr"r")r+r=rmr"r# Ssz4clean_state_dict_for_safetensors..rNrzRemoved shared tensor zk while saving. This should be OK, but check by verifying that you don't receive any warning while reloadingcSs*i|]\}}|t|tjr|n|qSr")rrTensor contiguous)r+kvr"r"r#rq[s*) collections defaultdictlistitemsrstrrr4setvaluesr8rnloggerwarning)rmptrsr=tensor shared_ptrs warn_namesrp found_namesr"rrr# clean_state_dict_for_safetensors<s&     rsave_on_each_nodesafe_serializationcCstjtjkr t|}|r ttddid}t|t rt |}nt j }tj r0|s0|||dStjr=|r?|||dSdSdS)a Save the data to disk. Use in place of `torch.save()`. Args: obj: The data to save f: The file (or file-like object) to use to save the data save_on_each_node (`bool`, *optional*, defaults to `False`): Whether to only save on the global main process safe_serialization (`bool`, *optional*, defaults to `False`): Whether to save `obj` using `safetensors` or the traditional PyTorch way (that uses `pickle`). formatpt)metadataN)r distributed_typerXLAxm_maybe_convert_to_cpursafe_save_filerrrrsaveis_main_processis_local_main_process)objfrr save_funcr"r"r#r_s    rz2.0.0z1.25.0c Ksz;trtj}d|vrd|d<tjtn|ddtj|fd|i|}Wtr:tj|r:tj||StrMtj|rNtj|www)a Compatible drop-in replacement of `torch.load()` which allows for `weights_only` to be used if `torch` version is 2.4.0 or higher. Otherwise will ignore the kwarg. Will also add (and then remove) an exception for numpy arrays Args: f: The file (or file-like object) to use to load the data map_location: a function, `torch.device`, string or a dict specifying how to remap storage locations **kwargs: Additional keyword arguments to pass to `torch.load()`. weights_onlyTN map_location) rr serializationget_safe_globalsadd_safe_globalsTORCH_SAFE_GLOBALSrcloadclear_safe_globals)rrkwargsold_safe_globals loaded_objr"r"r#rs&     rcCsHt|dst|dst|d|}t|dr|jSt|dr |jSt|S)z( Gets a pretty name from `obj`. __qualname____name__r*)rrdrrr|)rr"r"r#get_pretty_names   rcCs@|D]\}}t|tr||i}t||q|||<q|S)z Recursively merges two dictionaries. Args: source (`dict`): The dictionary to merge into `destination`. destination (`dict`): The dictionary to merge `source` into. )r{rdict setdefault merge_dicts)source destinationkeyvaluenoder"r"r#rs     rportcCsR|durd}ttjtj}|d|fdkWdS1s"wYdS)z Checks if a port is in use on `localhost`. Useful for checking if multiple `accelerate launch` commands have been run and need to see if the port is already in use. Ni._getattr.)rr)rrrr"r"r#recursive_getattrs r return_fqnsc Cs|s|nd|f}|g}g}|rS|}|r|\}}|D]$\}}t|tjjrA|r<|r2|d|n|} || |fq||q|rL|||fn|||s|dddS)aATraverse the model in bottom-up order and return the children modules in that order. Args: model (`torch.nn.Module`): the model to get the children of Returns: `list[torch.nn.Module]`: a list of children modules of `model` in bottom-up order. The last element is the `model` itself. rrN)rcr9rrr.rAr4) rFrtopstackordered_modulescurrent_modulecurrent_module_namer=r child_namer"r"r#get_module_children_bottom_ups&   r)TTF)FFr3)F)Wrxrrrcodecsrr functoolsrrtypesrtypingrnumpynprpackaging.versionrsafetensors.torchr rcommands.config.defaultr loggingr stater constantsr dataclassesrimportsrrrrrmodelingrtransformer_enginerversionsrrrtorch_xla.core.xla_modelcore xla_modelrr.rAboolr$r)r1r2rBrDrOrkrlrrr_corenp_core multiarray _reconstructndarraydtyperr4dtypes UInt32DTyperrrintrrrrr|rrzrr"r"r"r#sx                H U #$       (