o iΜ.@sddlmZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZddgZGdddeZd d ed e d ed ed ed e d e_deedeedeedeedeedeededBdedBdededeeBdeeBdeeBdededededed ed!df(d"d#ZdeedeedeedeedeedeededBdedBdededeeBdeeBdeeBdededededed ed!df(d$d%Z deedeedeedeedeedeededBdedBdededeeBdeeBdeeBdededededed ed!df(d&d'Z!e ed(  ) )    ) )d-deedeedeedeedeedeed*edBdeded+edBdedBdedBded ededeeBdeeBdeeBdededed!df,d,dZ"dS).)castN)Tensor)_capturable_doc_default_to_fused_or_foreach_device_dtype_check_for_fused_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc _fused_doc!_get_capturable_supported_devices_get_scalar_dtype _get_value _maximize_doc _params_doc_stack_if_compiling _to_scalar_use_grad_for_differentiable _view_as_real DeviceDictDeviceDtypeDict OptimizerParamsTAdamadamcseZdZ     ddddddddded eeBd eeeBeeBfd ed ed ededBdededededBdeddffddZfddZ ddZ e dddZ Z S)rMbP?g?g+?:0yE>rFN)foreachmaximize capturabledifferentiablefuseddecoupled_weight_decayparamslrbetaseps weight_decayamsgradrrr r!r"r#returnc st|tr|r | s td|dkrtdd|ks"td|d|ks-td|d|dkr9dksCntd |dd|dkrOdksYntd |dd|ksdtd |t|dtrrt|dtst|dtrt|dtstd t|dtr| s|rtd |ddkrtdt|dtr| s|rtd|ddkrtdttt|}|||||||| | | | d } t || | r| rt dd|_ |rt ddSdS)NElr as a Tensor is not supported for capturable=False and foreach=TruerTensor lr must be 1-elementzInvalid learning rate: zInvalid epsilon value: r?z#Invalid beta parameter at index 0: z#Invalid beta parameter at index 1: zInvalid weight_decay value: z0betas must be either both floats or both TensorszKbetas[0] as a Tensor is not supported for capturable=False and foreach=Truez!Tensor betas[0] must be 1-elementzKbetas[1] as a Tensor is not supported for capturable=False and foreach=Truez!Tensor betas[1] must be 1-element) r%r&r'r(r)rrr r!r"r#z)`fused` does not support `differentiable`Tz0`fused` and `foreach` cannot be `True` together.) isinstancer ValueErrornumelfloattuplemaprsuper__init__ RuntimeError_step_supports_amp_scaling)selfr$r%r&r'r(r)rrr r!r"r#defaults __class__R/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/torch/optim/adam.pyr6#s|       z Adam.__init__cst||jD]k}|dd|dd|dd|dd|dd|dd|dd}|d D]:}|j|g}t|d krst|d sst |d }|ds]|dritj |t |d |j d ntj |t d|d <q9q dS)Nr)Frrr r!r#r"r$rstepis_fuseddtypedevicerC) r5 __setstate__ param_groups setdefaultstategetlentorch is_tensorr2tensorr rD)r9rIgroupr"pp_statestep_valr;r=r>rFss4           zAdam.__setstate__c Cs~d}|dD]} | jdur|t| O}|| | jjr!td|| j|j| } t| dkr||dr:t| |dsB|drPtj dt |dd| j d ntj d t d | d <tj | tjd | d<tj | tjd | d<|dr|tj | tjd | d<|| d|| d|dr|| d|dr| d jrtd|drt|dr|dstd|| d q|S)NFr$zJAdam does not support sparse gradients, please consider SparseAdam insteadrr"r r=r@rBr-rEr?) memory_formatexp_avg exp_avg_sqr)max_exp_avg_sqr!zB`requires_grad` is not supported for `step` in differentiable moderr%r+)gradrL is_complexappend is_sparser7rIrKrzerosr rDrN zeros_likepreserve_format requires_gradrM) r9rOparams_with_gradgradsexp_avgs exp_avg_sqsmax_exp_avg_sqs state_steps has_complexrPrIr=r=r> _init_groupsl             zAdam._init_groupc Cs|d}|dur!t |}Wdn1swY|jD]V}g}g}g}g}g}g} |d\} } |||||||| } t|||||| f|d| | | |d|d|d|d|d|d |d |d t|d dt|d d|ddq$|S)zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nr&r)r%r(r'rrr r!r" grad_scale found_infr#)r)rebeta1beta2r%r(r'rrr r!r"rgrhr#) _cuda_graph_capture_health_checkrL enable_gradrGrfrgetattr) r9closurelossrOr_r`rarbrcrdrirjrer=r=r>r?s`       z Adam.step)rrrrFN)__name__ __module__ __qualname__rr2rr3boolr6rFrfrr? __classcell__r=r=r;r>r"sX       PKaf Implements Adam algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)}, \: \textit{amsgrad}, \:\textit{maximize}, \: \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0\leftarrow 0 \text{ (second moment)},\: v_0^{max}\leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm} v_t^{max} \leftarrow \mathrm{max}(v_{t-1}^{max},v_t) \\ &\hspace{10mm}\widehat{v_t} \leftarrow v_t^{max}/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. z Args: aG lr (float, Tensor, optional): learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True. betas (tuple[Union[float, Tensor], Union[float, Tensor]], optional): coefficients used for computing running averages of gradient and its square. If a tensor is provided, must be 1-element. (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) decoupled_weight_decay (bool, optional): if True, this optimizer is equivalent to AdamW and the algorithm will not accumulate weight decay in the momentum nor variance. (default: False) amsgrad (bool, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) z a= .. Note:: A prototype implementation of Adam and AdamW for MPS supports `torch.float32` and `torch.float16`. .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ r$r`rarbrcrdrgrhr)rerirjr%r(r'rr r!r#r*c 'Cs|dus|dur tdtjr_single_tensor_adam[s                       rc + st|dkrdSttr|stddkrtdttr2|s(tddkr2tdttrG|s=tddkrGtdtjsk|rkt d d t fd d t ||d dDskt dd|duss|durwt d|r}t dt t t t||||||g}ttrtjdkrjind}|D]\\}}}}}}}ttt|}ttt|}ttt|}ttt|}ttt|} |dj}!|dur|!|vrj|!d d||!<|r||!n}"| r|r ttt|}#t|||||#nt|||||rt|}tjs6| djr6tj| tjdddddnt| d| dkrc|rOt|d| n|r[tj||| dntj||| d}t||ttd|"t|ttjrt|d}$d}%n|}$d}%t ||$||%~~$|rt!| }&t!| }'t"|&dt"|'dt#|'t$|&t%|&t&|'|&}(|'})|rttt|}#t'|#|t(|#}*nt(|}*t$|*|)t|*|t$|*|(t)|||*qfdd| D}&fdd| D}'t*fdd|&D}(dd|'D})|r:ttt|}#t'|#|t(|#}*nt(|}*t$|*|)t|*|t)|||*|(qdS)Nrr+rr,zHbeta1 as a Tensor is not supported for capturable=False and foreach=TruezTensor beta1 must be 1-elementzHbeta2 as a Tensor is not supported for capturable=False and foreach=TruezTensor beta2 must be 1-elementF) supports_xlac3s0|]\}}|jj|jjko|jjvVqdSrp)rDr).0rPr?)rr=r> `s   z%_multi_tensor_adam..T)strictrwrxrvz#_foreach ops don't support autogradcpurDr{r.)rDrycg|] }dt|qSrrrr?)rir=r> z&_multi_tensor_adam..crrrr)rjr=r>rrcsg|]}|dqS)r=rbc)r%r=r>r scSsg|]}|dqS)r~r=rr=r=r>r s)+rKr/rr7r1r0rLrrr allziprrr"_group_tensors_by_device_and_dtypestrrDvaluesrlistrr _foreach_negis_cpu _foreach_add_rN _foreach_mul_ _foreach_add_foreach_lerp_r2 _foreach_mul_foreach_addcmul_ _foreach_pow _foreach_sub_ _foreach_neg_ _foreach_div__foreach_reciprocal__foreach_sqrt__foreach_maximum_ _foreach_sqrt_foreach_addcdiv_r)+r$r`rarbrcrdrgrhr)rerirjr%r(r'rr r!r#grouped_tensorsrdevice_params_ device_grads_device_exp_avgs_device_exp_avg_sqs_device_max_exp_avg_sqs_device_state_steps__ device_params device_gradsdevice_exp_avgsdevice_exp_avg_sqsdevice_state_stepsrDrdevice_max_exp_avg_sqsscaled_device_gradsr}rrrrexp_avg_sq_sqrtr=)rirjrr%r>_multi_tensor_adam*s0                                      rc 'Cs|sdS|r tdt| } t| } |dur|j|ini}|dur&|j|ini}t| tr9t| jdkr9| j| ind}t||||||g}|D]\\}}\\}}}}}}}t t t|}t t t|} t t t|}!t t t|}"t t t|}#d\}$}%|dur| ||j |dd}$|dur| ||j |dd}%|dur||vr| j |dd||<||} t |#d|st jnt j}&|&|| |!|"||#|| | | | |||$|%d|%durt |#|%gt|#qJdS) Nz9Adam with fused=True does not support differentiable=Truer)NNT)r{rr) r)r%rirjr(r'rrgrh)r7rrDr/rrrritemsrrrHrrLr _fused_adam_ _fused_adamw_rrK)'r$r`rarbrcrdrgrhr)rerirjr%r(r'rr r!r#grad_scale_dictfound_inf_dictlr_dictrrDrrrrrrrrrrrrdevice_grad_scaledevice_found_inffuncr=r=r> _fused_adam#s$  r)single_tensor_fnFrr"cCs| dur|durt||dd\}}|rt|tr|sd}| dur"d} |dur(d}tjs:tdd|Ds:td|rEtj rEtd| rPtj rPtd| rZtj sZt }n |rdtj sdt }nt }|||||||f|| ||||||||| | | d dS) znFunctional API that performs Adam algorithm computation. See :class:`~torch.optim.Adam` for details. NF) use_fusedcss|] }t|tjVqdSrp)r/rLr)rtr=r=r>rs  zadam..zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsz6torch.jit.script not supported with foreach optimizersz4torch.jit.script not supported with fused optimizers) r)rerirjr%r(r'rr r!rgrhr#) rr/rrLrrrr7rrrrr)r$r`rarbrcrdrr r!r"rgrhrer#r)rirjr%r(r'rrrr=r=r>rs^#  )NFFNNNFF)#typingrrLr optimizerrrrrr r r r r rrrrrrrrrrr__all__r__doc__rrtr2rrrrr=r=r=r>s  Xs%H     P     z     c