o ‚›ix3ã@s†ddlZddlZddlmZddlZddlmZmZddlmZddl m Z ddl m Z e   e¡ZedƒGd d „d ejƒƒZeZed ƒGd d „d ejƒƒZedƒGdd„dejƒƒZedƒGdd„dejƒƒZedƒGdd„dejƒƒZedƒGdd„dejƒƒZGdd„dejƒZGdd„dejƒZGdd„dejƒZGd d!„d!ejƒZGd"d#„d#ejƒZGd$d%„d%ejƒZGd&d'„d'eƒZGd(d)„d)ejƒZid*e“d+ed,d-d.œf“d/e“d0e“d1ed2d3if“d4e“d5ed6d3if“d7e“d8e“d9ej “d:e“d;e“de“d?ej"“d@ej#“eej$ej%ej&edAœ¥Z'ee'ƒZ(dBdC„Z)e)d1ƒZ*e)d0ƒZ+e)d*ƒZ,e)d/ƒZ-e)d<ƒZ.e)dDƒZ/e)d;ƒZ0e)d:ƒZ1dS)EéN)Ú OrderedDict)ÚTensorÚnné)Úuse_kernel_forward_from_hub)Úlogging)Úis_torchdynamo_compilingÚGeluTanhcóLeZdZdZd def‡fdd„ Zdedefdd „Zdedefd d „Z‡Z S) ÚGELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. FÚuse_gelu_tanh_pythoncs2tƒ ¡|r |j|_dStjtjjdd|_dS)NÚtanh)Ú approximate) ÚsuperÚ__init__Ú_gelu_tanh_pythonÚactÚ functoolsÚpartialrÚ functionalÚgelu)Úselfr ©Ú __class__©úZ/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/activations.pyr(s  zGELUTanh.__init__ÚinputÚreturnc Cs6|ddt t dtj¡|dt |d¡¡S©Nçà?çð?ç@ç÷Hmâä¦?g@©Útorchr ÚmathÚsqrtÚpiÚpow©rrrrrr/ó6zGELUTanh._gelu_tanh_pythoncCó | |¡S©N©rr)rrrÚforward2ó zGELUTanh.forward©F) Ú__name__Ú __module__Ú __qualname__Ú__doc__Úboolrrrr.Ú __classcell__rrrrr s r ÚNewGELUc@ó"eZdZdZdedefdd„ZdS)ÚNewGELUActivationzÎ Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 rrc Cs6d|dt t dtj¡|dt |d¡¡Srr#r)rrrr.Ar*zNewGELUActivation.forwardN©r1r2r3r4rr.rrrrr9:sr9ÚGeLUcr ) ÚGELUActivationa³ Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 FÚuse_gelu_pythoncs(tƒ ¡|r |j|_dStjj|_dSr,)rrÚ _gelu_pythonrrrr)rr=rrrrNs  zGELUActivation.__init__rrcCs |ddt |t d¡¡S)Nrr r!)r$Úerfr%r&r)rrrr>Us zGELUActivation._gelu_pythoncCr+r,r-r)rrrr.Xr/zGELUActivation.forwardr0) r1r2r3r4r5rrr>r.r6rrrrr<Es r<ÚSiLUc@r8)ÚSiLUActivationaè See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rrcCs tj |¡Sr,)rrÚsilur)rrrr.fs zSiLUActivation.forwardNr:rrrrrA\srAÚFastGELUc@r8)ÚFastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rrcCs*d|dt |ddd||¡S)Nrr g€ÑÓ3Eˆé?r")r$r r)rrrr.ps*zFastGELUActivation.forwardNr:rrrrrDjórDÚ QuickGELUc@r8)ÚQuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rrcCs|t d|¡S)Ng¬Zd;û?)r$Úsigmoidr)rrrr.zszQuickGELUActivation.forwardNr:rrrrrGtrErGcs<eZdZdZdedef‡fdd„ Zdedefdd „Z‡ZS) ÚClippedGELUActivationa’ Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 ÚminÚmaxcs8||krtd|›d|›dƒ‚tƒ ¡||_||_dS)Nzmin should be < max (got min: z, max: ú))Ú ValueErrorrrrJrK)rrJrKrrrr‹s   zClippedGELUActivation.__init__ÚxrcCst t|ƒ|j|j¡Sr,)r$ÚcliprrJrK)rrNrrrr.“ózClippedGELUActivation.forward) r1r2r3r4Úfloatrrr.r6rrrrrI~s rIcs2eZdZdZ‡fdd„Zdedefdd„Z‡ZS)ÚAccurateGELUActivationzÙ Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cs tƒ ¡t dtj¡|_dS)Né)rrr%r&r'Úprecomputed_constant©rrrrrŸs zAccurateGELUActivation.__init__rrc Cs,d|dt |j|dt |d¡¡S)Nrrr"é)r$r rTr(r)rrrr.£s,zAccurateGELUActivation.forward)r1r2r3r4rrr.r6rrrrrR—s rRcsDeZdZdZ‡fdd„Zdedefdd„Zdedefdd „Z‡ZS) ÚMishActivationzÙ See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish cstƒ ¡tjj|_dSr,)rrrrÚmishrrUrrrr­s zMishActivation.__init__rrcCs|t tj |¡¡Sr,)r$r rrÚsoftplusr)rrrÚ _mish_python±rPzMishActivation._mish_pythoncCr+r,r-r)rrrr.´r/zMishActivation.forward) r1r2r3r4rrrZr.r6rrrrrW§s  rWc@r8)ÚLinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. rrcCs|Sr,rr)rrrr.½szLinearActivation.forwardNr:rrrrr[¸sr[c@seZdZdZddd„ZdS)ÚLaplaceActivationzû Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹øÛž æ?ç ^×/Ø Ò?cCs*|| |t d¡¡}ddt |¡S)Nr!rr )Údivr%r&r$r?)rrÚmuÚsigmarrrr.ÉszLaplaceActivation.forwardN)r]r^©r1r2r3r4r.rrrrr\Ásr\c@seZdZdZdd„ZdS)ÚReLUSquaredActivationz^ Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668 cCstj |¡}t |¡}|Sr,)rrÚrelur$Úsquare)rrÚ relu_appliedÚsquaredrrrr.Ós  zReLUSquaredActivation.forwardNrbrrrrrcÎs rccseZdZ‡fdd„Z‡ZS)ÚClassInstantiercs4tƒ |¡}t|tƒr |n|if\}}|di|¤ŽS)Nr)rÚ __getitem__Ú isinstanceÚtuple)rÚkeyÚcontentÚclsÚkwargsrrrriÚs zClassInstantier.__getitem__)r1r2r3rir6rrrrrhÙsrhcsfeZdZdZddddejdf‡fdd„ Zded efd d „Zded efd d „Z ded efdd„Z ‡Z S)ÚXIELUActivationzî Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python gš™™™™™é?rgíµ ÷Æ°¾Fc sntƒ ¡t t t tj||d¡¡ d¡¡|_ t t t tj|||d¡¡ d¡¡|_ |  dtj||d¡|  dtj||d¡||_ t |ƒ|_t |ƒ|_d|_zFddl}tjj ¡|_d}zddlm} | |jƒ|_|d7}Wnty“} z|d| ›d 7}|j|_WYd} ~ nd} ~ wwt |¡WdSty¶} zt d t| ƒ¡WYd} ~ dSd} ~ ww) N)ÚdtyperÚbetaÚepszUsing experimental xIELU CUDA.)Úallow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.u¡CUDA-fused xIELU not available (%s) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)rrrÚ Parameterr$ÚlogÚexpm1ÚtensorÚ unsqueezeÚalpha_pÚalpha_nÚregister_bufferÚwith_vector_loadsrQÚ _beta_scalarÚ _eps_scalarÚ_xielu_cuda_objÚ xielu.opsÚclassesÚxieluÚXIELUÚtorch.compilerrtÚ _xielu_cudaÚ_xielu_cuda_fnÚ ExceptionÚloggerÚ warning_onceÚstr) rÚ alpha_p_initÚ alpha_n_initrrrsrqr}rƒÚmsgrtÚerrrrrrès@ ("ÿ     €þý€ÿzXIELUActivation.__init__rNrc Cshtj |j¡}|jtj |j¡}t |dk||||j|t t  ||j ¡¡|||j|¡S)Nr) rrrYrzrrr{r$ÚwhererwrJrs)rrNrzr{rrrÚ _xielu_pythons$ýzXIELUActivation._xielu_pythoncCsœ|j}| ¡dkr| d¡}| ¡dks | ¡dkr$| dd| d¡¡}||jkr1t d||j¡|j ||j   |j ¡|j   |j ¡|j |j|j¡}| |¡S)zDFirewall function to prevent torch.compile from seeing .item() callsrVréÿÿÿÿrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)ÚshapeÚdimryÚviewÚsizer‰rŠr€r.rzÚtorqr{r~rr})rrNÚoriginal_shapeÚresultrrrr†s*   ÿ  ý  ù zXIELUActivation._xielu_cudarcCs4|jdur|jrtƒs| |¡St d¡| |¡S)Nz:torch._dynamo is compiling, using Python version of xIELU.)r€Úis_cudarr‡r‰rŠr‘r)rrrr.5s    zXIELUActivation.forward) r1r2r3r4r$Úbfloat16rrr‘r†r.r6rrrrrpàs ù+ rprÚgelu_10iöÿÿÿé )rJrKÚ gelu_fastÚgelu_newÚ gelu_pythonr=TÚgelu_pytorch_tanhÚgelu_python_tanhr Ú gelu_accurateÚlaplaceÚ leaky_reluÚlinearrXÚ quick_gelurdÚrelu2Úrelu6rH)rBÚswishr ÚprelurƒcCs,|tvrt|Std|›dtt ¡ƒ›ƒ‚)Nz function z not found in ACT2FN mapping )ÚACT2FNÚKeyErrorÚlistÚkeys)Úactivation_stringrrrÚget_activationYsr±rB)2rr%Ú collectionsrr$rrÚintegrations.hub_kernelsrÚutilsrÚutils.import_utilsrÚ get_loggerr1r‰ÚModuler ÚPytorchGELUTanhr9r<rArDrGrIrRrWr[r\rcrhrpÚ LeakyReLUÚReLUÚReLU6ÚSigmoidr@ÚTanhÚPReLUÚACT2CLSr¬r±r rŸrržr§rBrXÚ linear_actrrrrÚs¢            ^ÿþýü ûú ùø ÷ ö õ ô óòñðïê