o iAA@sZUddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z ddlmZdd lmZe eZzQdd lmZmZmZmZmZdd lmZdd lmZzdd lmZWn e ygdZYnwz ddlm!Z"dZ#Wn e y}dZ#Ynwej$%dd&Z'dZ(e'e vZ)de*fddZde*fddZ!dedddidedddiej+edd diej+edd diej+ed!d"diej+ed#d"diej+edd did$ded%d&diej,ed'd(dej+ed'd(diej+ed)d(diej+ed'd(did*dej+ej-Bed+d,dd-iidej+ej-Bed+d.dd-iidej+ej-Bed+d/dd-iidej+ej-Bed+d0dd-iidej+ej-Bed+d1dd-iidej+ej-Bed+d2dd-iid3 Z.e/e*e/ee*Bee/eefBffe0d4<edurej+ed5d6d7iej+ed5d6d7id8e.d9<d:d;Z1d`ddZd?dZ!Gd@dAdAZdBdCZdDdEZdFd=Z2YnwdGddHdIddHdIddHdJZ3e/e*e/e*e*ffe0dK<iZ4e/e*edBfe0dL<dMe*dBdNe5fdOdPZ6 d`dMe*dQedBdNedBfdRdSZ7e4fdTe*dUe/e*edBffdVdWZ8dadTe*dXe*dBdYe9e*BdBdNefdZd[Zd\e:eeBfd]d^Z;gd_ZEz-use_kernel_forward_from_hub..)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)rrrrr>s r func_namecCs>trtrt|StstdddStdtddS)Nzwuse_kernel_func_from_hub is not available in the installed kernels version. Please upgrade kernels to use this feature.rcSrrrfuncrrrrTrz*use_kernel_func_from_hub..)r _has_use_kernel_func_from_hub!_kernels_use_kernel_func_from_hubr"r#r$)r%rrrrGsrcudaz!kernels-community/deformable-detrMultiScaleDeformableAttention)repo_idrzkernels-community/moe Llama4TextMoezkernels-community/liger_kernels LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)r*rocmxpumpsnpuzmedmekk/triton-llama-mlpTritonLlamaMLPzkernels-community/megablocksMegaBlocksMoeMLPzahadnagy/megablocks)r*r0r1zkernels-community/activationFastGELU)r,rr QuickGELUNewGELUSiluGeluGeluTanh) r+r-r/MLPr5r6r7r8SiLUGeLUr;_KERNEL_MAPPINGzkernels-community/rotaryapply_rotary_transformers)r,r%)r1r*rotary_pos_embcs"|vptfdd|DS)Nc3s$|] }t|to t|VqdSr) isinstancedicthas_key).0vkeyrr s"zhas_key..)anyvalues)drHrrGrrDs"rDcCs4|durt}t|drtddstdt|dS)Nr1z0.10.2) MIN_VERSIONzfkernels uses an incompatible version. Please install the latest version with `pip install -U kernels`.)r?rDr ImportErrorr)mappingrrr$register_kernel_mapping_transformerss rPcO dd}|S)NcSrrrrrrr decoratorz.use_kernel_forward_from_hub..decoratorrargskwargsrRrrrrcOrQ)NcSrrrr&rrrrRrSz+use_kernel_func_from_hub..decoratorrrTrrrrrWc@seZdZddZdS)r cOtd)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeError)selfrUrVrrr__init__zLayerRepository.__init__N)__name__ __module__ __qualname__r\rrrrr s r cOrX)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rYrUrVrrrrrcOrX)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.rYrarrrr r]rcOrX)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.rYrarrrrPrbzkernels-community/causal-conv1d)r,rzkernels-community/mamba-ssm)z causal-conv1dz mamba-ssmzfalcon_mamba-ssm_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationreturncCs|duo td|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rerrr is_kernelsriattention_wrapperc Cs0ddlm}ddlm}d|vr|ddn|}t|sdSts%tdd|vr5|d\}}|}n|}d}|}| d \}}}|}|rO|nd}zt ||d } Wnt yp} z t d |d | d d} ~ wwt | dr|dur|t}|} n |durt| |} ||| |||d| S)a< Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continious batching, where we want the `paged` wrapper, which calls the paged cache. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|r Nz`kernels` is either not installed or uses an incompatible version. Please install the latest version with `pip install -U kernels`.:@)revisionz-An error occurred while trying to load from 'z': .flash_attn_varlen_funcflash_attention_2) masking_utilsrkmodeling_utilsrlsplitri_kernels_availablerNstrip partitionr Exception ValueErrorhasattrr getattrregister) rerjrkrlactual_attn_namer, kernel_name_revkernelekernel_functionrrrload_and_register_attn_kernel%sB     rrrOc Cs||vrt||tr||S|tvr"td|dd||<dStrjz&t|d}t|dd}t|dd}t|||d}|||<W||StyZd||<Y||St yid||<Y||Swddl }| dd }d |d }z| d t } t| |d} Wn tyd} Ynwt| r| rz | |} | ||<| WStyd||<Y||Swd||<||S) NzKernel z! not found in _HUB_KERNEL_MAPPINGr,rprrprr-ris_ _availablez..utils.import_utils)rBrrcr"r#rwgetrFileNotFoundErrorAssertionError importlibreplace import_module __package__r}rzcallable) rrOr,rprrrnew_kernel_namer% utils_modis_kernel_availablemodulerrrlazy_load_kernel_sR           rrprcCsbddlm}d||d}tr-tjd}t|tdkr&t||||dSt|||dSt d ) Nr) __version__ transformers) frameworkrr,kernelsz0.10.4)rpr user_agentrzFkernels is not installed, please install it with `pip install kernels`) rrwrmetadatar pkg_versionparseget_kernel_hubrN)rrprrrkernels_versionrrrrs   r module_namescs ttrgfdd}|S)a This decorator attaches the target function as an attribute of the module. The function must already be decorated with @use_kernel_func_from_hub this decorator then wraps it as an nn.Module internally. When kernelize is later applied to the full model, the function can be accessed as a regular module attribute and kernelized just like any other layer. The kernelization is performed in place, modifying the module directly. cs|jfdd}||_|S)Ncs0|g|Ri|D]}t|d|q dS)N rotary_fn)setattr)r[rUrVfn)r orig_initrrnew_initsz8use_kernelized_func..decorator..new_init)r\)rrr)rrrRsz&use_kernelized_func..decorator)rBr)rrRrrruse_kernelized_funcs   r) r rrrrPrrrrr)NN)=importlib.metadatarosrgcollections.abcrtypesr packagingrrutilsrrutils.import_utilsrflash_attentionr get_loggerr^r"rr r r rrrrrr!rrNrr)r(environrupperr$rwr str INFERENCETRAINING TORCH_COMPILEr?rC__annotations__rDrPrcrdboolrirrintlistr__all__rrrrs\             "       .     %  ":(/