o i0(@sJddlmZmZerddlZddZddZdd ZGd d d eZdS) )PushToHubMixinis_torch_availableNcCsVd}zt|}Wntytd|w|jj}|dkr)tjjdur)dS|S)z Infers the device type from the model parameters. Args: model: The model instance. Returns: The device type. z { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } zPCannot determine model device, please provide a device to the mapping. Example: cudaNrocm) next parameters StopIteration ValueErrordevicetypetorchversionhip)modelEXAMPLE_MAPPINGparamdev_typerb/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/utils/kernel_config.py infer_devices    rcCsZddlm}|dvrtd||dd}|dd}|||||dii||<dS)NrLayerRepositoryrrxpunpu5Only cuda, rocm, xpu and npu devices supported, got: :)repo_id layer_name)kernelsrr split)r r repo_namemodecompatible_mappingrrepo_layer_namerrrradd_to_mapping9s  r'c Cszddlm}ddlm}|dvrtd||dd}|dd}|dd } |||||| |d ii||<dS) Nr)Path)LocalLayerRepositoryrrrr/) repo_path package_namer )pathlibr(r!r)r r") r r r#r$r%r(r)r&r,repo_package_namerrradd_to_mapping_localJs   r0c@sBeZdZdZidfddZdddZdd Zd d Zdd d ZdS) KernelConfigze Kernel configuration class. This class is used to configure the kernel mapping for a model. FcCs||_i|_||_dSN)kernel_mappingregistered_layer_namesuse_local_kernel)selfr3r5rrr__init__ds zKernelConfig.__init__NcCs,ddlm}||||||dii|j|<dS)Nrr)rr revision)r!rr3)r6rregistered_namer r r$r8rrrr update_kernelis zKernelConfig.update_kernelcCs,|D]\}}t|dr|j|j|<qdS)Nkernel_layer_name) named_moduleshasattrr;r4)r6rnamemodulerrrstore_registered_layer_namesvs   z)KernelConfig.store_registered_layer_namescCsd}||t|jtstd|dt|j|jD]e\}}||jvr2td|dt|t rJd|vs?d|vrItd|d |qt|tr{|D]&\}}|d vrbtd |t|t rod|vsod|vrytd|d |qSqtd |d|d S)aJ Validates the kernel_mapping to ensure that: 1. Each layer_name in the mapping is registered in the model (i.e., the model contains a module with a matching kernel_layer_name). 2. Each kernel value is either a string of the form 'org/repo:layer_name' or a dict mapping device types ("cuda", "rocm", "xpu", "npu") to such strings. 3. Each device key in a dict is one of "cuda", "rocm", "xpu", or "npu". 4. Each repo_name is a valid repository and layer name in the format 'org/repo:layer_name' (i.e., a string containing both a slash and a colon). 5. If a local path is detected, it should be in the format '/abs/path:layer_name'. The absolute path must include the `package_name`, like "/home/user/layer_norm". Args: model: The model instance whose modules are checked for registered kernel_layer_name attributes. Raises: ValueError: If a layer_name is not registered in the model, if a device is not supported, or if a repo_name is not a valid 'org/repo:layer_name' string. a For single device form remote { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, For multiple devices form remote { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", "rocm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } For single device form local { "RMSNorm": "/abs/path:LlamaRMSNorm", ... }, For multiple devices form local { "RMSNorm": { "cuda": "/abs/path:LlamaRMSNorm", "rocm": "/abs/path:LlamaRMSNorm", ... }, ... } z7Kernel mapping must be a dict of the following format: z, got: zLayer z` is not registered in the model, please register it first using register_kernel_forward_from_hubr*rzKernel mapping for 'zl' must be a valid repo name with a layer name (e.g., 'org/repo:layer_name' or '/abs/path:layer_name'), got: rrz'Kernel mapping must follow the format: N) r@ isinstancer3dictr r itemsr4valuesstr)r6rMAPPING_FORMATr kernelr r#rrrsanitize_kernel_mapping{s: $     z$KernelConfig.sanitize_kernel_mappingc Csddlm}i}t|}|jD]W\}}|jr|jn|j}|r%||jB}t |t rA|} |j s8t ||| ||qt ||| ||qt |trh|D]\} } | |krSqJ|j s_t || | ||qJt || | ||qJq||_dS)aW Transforms a simple kernel_mapping of the form: { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, or for local path: { "RMSNorm": "/home/user/liger_kernels:LigerRMSNorm", ... }, into a nested mapping: { "RMSNorm": { "cuda": { Mode.INFERENCE: LayerRepository( repo_id="kernels-community/layer_norm", layer_name="LlamaRMSNorm", ) } } } or for local path: { "RMSNorm": { "cuda": { Mode.INFERENCE: LocalLayerRepository( repo_path=Path("/home/user/liger_kernels"), package_name="liger_kernels", layer_name="LigerRMSNorm", ) } } } that's compatible with the kernels library. The device is inferred from the model's parameters if not provided. The Mode is inferred from the model's training state. r)ModeN)r!rIrr3rCtrainingTRAINING INFERENCE TORCH_COMPILErArEr5r'r0rB) r6rcompilerIr%current_devicer rGr$r#r rrrcreate_compatible_mappings* 1    z&KernelConfig.create_compatible_mappingr2)F) __name__ __module__ __qualname____doc__r7r:r@rHrPrrrrr1_s  Sr1)utilsrrr rr'r0r1rrrrs#