o 0i9@sddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZd d Zd dZGdddZGdddZGdddeZGddde ZGdddZGdddZdS)N)deepcopy)optim)AcceleratedOptimizer)AcceleratedScheduler)DistributedType)is_bnb_available)compare_versionsc Csdd|jD}ddlm}|}tdddrUd|d <t|tj}trO|sOdd l m}t||j|j frMz|j d k}Wnt yL|j j d k}Ynwd}|rUd |d <tddd rt|tj}tr|sdd l m}t||j|jfrz|j d k}Wnt y|j j d k}Ynw|rddlm}|}tddrtdddrddl m}m} t||| frz|j d k} Wnt y|j j d k} Ynw| rddlm} | }||jfi|S)z Args: optimizer: torch.optim.Optimizer Returns the DeepSeedCPUOptimizer (deepspeed.ops) version of the optimizer. cSsi|] \}}|dvr||qS))lr weight_decay).0kvr r \/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/accelerate/utils/deepspeed.py %sz2map_pytorch_optim_to_deepspeed..r)DeepSpeedCPUAdam deepspeedz>=z0.3.1F adamw_modeN Tz0.5.5)DeepSpeedCPUAdagradz0.38.0) min_versionz0.11.0)Lion Lion32bit)DeepSpeedCPULion)defaultsitemsdeepspeed.ops.adamrr isinstancerAdamWr bitsandbytes.optim AdamW32bit optim_bitsAttributeErrorargsAdagrad Adagrad32bitdeepspeed.ops.adagradrrrdeepspeed.ops.lionr param_groups) optimizerrroptimizer_classis_adawbnb_optis_adarrr is_bnb_32bitsrr r rmap_pytorch_optim_to_deepspeedsT              r1cCs>|jtjkr tdt|jts|jStdd|jDS)z Returns the currently active DeepSpeedPlugin. Raises: ValueError: If DeepSpeed was not enabled and this function is called. a!Couldn't retrieve the active `DeepSpeedPlugin` as none were enabled. Please make sure that either `Accelerator` is configured for `deepspeed` or make sure that the desired `DeepSpeedPlugin` has been enabled (`AcceleratorState().select_deepspeed_plugin(name)`) before calling this function.css|]}|jr|VqdSN)selected)rpluginr r r tsz.get_active_deepspeed_plugin..) distributed_typer DEEPSPEED ValueErrorrdeepspeed_pluginsdictnextvalues)stater r rget_active_deepspeed_pluginds  r>c@sdeZdZdZddZddZddZdd d Zdd d ZddZ ddZ ddZ ddZ ddZ dS)HfDeepSpeedConfigaJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. c Cst|tr t|}nUtj|r,t|dd }t|}Wdn1s&wYn3z!zt |}Wntj yKt | d}t |}YnwWntttfy^td|w||_|dS)Nzutf-8)encodingzoExpected a string path to an existing deepspeed config, or a dictionary, or a base64 encoded string. Received: )rr:rospathexistsopenjsonloadloadsJSONDecodeErrorbase64urlsafe_b64decodedecodeUnicodeDecodeErrorr$r8configset_stage_and_offload)selfconfig_file_or_dictrMfconfig_decodedr r r__init__s,     zHfDeepSpeedConfig.__init__cCsj|dd|_d|_|s|r3tddg}t|d|dg}t||@dkr1d |_dSdSdS) Nzzero_optimization.stageFcpunvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicerT) get_value_stage_offloadis_zero2is_zero3setlen)rOoffload_devices_validoffload_devicesr r rrNs  z'HfDeepSpeedConfig.set_stage_and_offloadcCsH|j}|d}|}|D]}||}|durd|fSq||fS)N.)rMsplitpopget)rO ds_key_longrMnodesds_keynoder r rfind_config_nodes   z"HfDeepSpeedConfig.find_config_nodeNcCs&||\}}|dur |S|||S)zG Returns the set value or `default` if no value is set N)rhrc)rOrddefaultrMrfr r rrWs zHfDeepSpeedConfig.get_valueFcCsj|j}|d}|D]}|}||}|dur'|r$td|d|jdSq |dur3||dSdS)z Deletes a sub-section of the config file if it's found. Unless `must_exist` is `True` the section doesn't have to exist. r`Nz Can't find z entry in the config: )rMrarcr8rb)rOrd must_existrMrerg parent_configr r rdel_config_sub_trees  z%HfDeepSpeedConfig.del_config_sub_treecCs||}|dur dSt|S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set). NFrWboolrOrdvaluer r ris_trues zHfDeepSpeedConfig.is_truecCs ||}|dur dSt| S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set). NFrmror r ris_falses zHfDeepSpeedConfig.is_falsecC |jdkS)NrrXrOr r rrZ zHfDeepSpeedConfig.is_zero2cCrs)Nrtrur r rr[rvzHfDeepSpeedConfig.is_zero3cCs|jSr2)rYrur r r is_offloadszHfDeepSpeedConfig.is_offloadr2)F)__name__ __module__ __qualname____doc__rSrNrhrWrlrqrrrZr[rxr r r rr?ws   r?c@s*eZdZdZddZd ddZddZd S) DeepSpeedEngineWrapperz Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop. Args: engine (deepspeed.runtime.engine.DeepSpeedEngine): deepspeed engine to wrap cCs ||_dSr2)engine)rOr~r r rrSrvzDeepSpeedEngineWrapper.__init__TcKs8|jj|d|jj|fi||r|jdSdS)N) is_boundary)r~"set_gradient_accumulation_boundarybackwardstep)rOlosssync_gradientskwargsr r rrs zDeepSpeedEngineWrapper.backwardcCs |j}t|dr|S|S)z3Get the global gradient norm from DeepSpeed engine.item)r~get_global_grad_normhasattrr)rO grad_normr r rrs  z+DeepSpeedEngineWrapper.get_global_grad_normN)T)ryrzr{r|rSrrr r r rr}s   r}cs>eZdZdZfddZd ddZddZed d ZZ S) DeepSpeedOptimizerWrapperz Internal wrapper around a deepspeed optimizer. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. cs$tj|dddt|jd|_dS)NF)device_placementscaleroverflow)superrSrr+__has_overflow__)rOr+ __class__r rrS0sz"DeepSpeedOptimizerWrapper.__init__NcCdSr2r )rO set_to_noner r r zero_grad4z#DeepSpeedOptimizerWrapper.zero_gradcCrr2r rur r rr7rzDeepSpeedOptimizerWrapper.stepcCs|jr|jjSdS)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.F)rr+rrur r rstep_was_skipped:sz*DeepSpeedOptimizerWrapper.step_was_skippedr2) ryrzr{r|rSrrpropertyr __classcell__r r rrr's  rcs(eZdZdZfddZddZZS)DeepSpeedSchedulerWrapperz Internal wrapper around a deepspeed scheduler. Args: scheduler (`torch.optim.lr_scheduler.LambdaLR`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): cst||dSr2)rrS)rO scheduler optimizersrr rrSLsz"DeepSpeedSchedulerWrapper.__init__cCrr2r rur r rrOrzDeepSpeedSchedulerWrapper.step)ryrzr{r|rSrrr r rrrBs rc@seZdZdZdddZdS) DummyOptima Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file. Args: lr (float): Learning rate. params (iterable): iterable of parameters to optimize or dicts defining parameter groups weight_decay (float): Weight decay. **kwargs (additional keyword arguments, *optional*): Other arguments. MbP?rcKs||_||_||_||_dSr2)paramsr r r)rOrr r rr r rrScs zDummyOptim.__init__N)rrryrzr{r|rSr r r rrSsrc@seZdZdZdddZdS)DummySchedulera Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int, *optional*): Total number of steps. warmup_num_steps (int, *optional*): Number of steps for warmup. lr_scheduler_callable (callable, *optional*): A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`. **kwargs (additional keyword arguments, *optional*): Other arguments. NrcKs"||_||_||_||_||_dSr2)r+total_num_stepswarmup_num_stepslr_scheduler_callabler)rOr+rrrrr r rrS|s  zDummyScheduler.__init__)NrNrr r r rrjsr)rIrErAcopyrtorchrr+rrr dataclassesrimportsr versionsr r1r>r?r}rrrrr r r rs&       G*