o 0ia@s\ddlmZddlmZddlmZmZddlmZdddZ d d Z dd d Z ddZ dS)) MethodTypeN)is_hpu_availableis_transformer_engine_available)GatheredParametersTc Cststdtrddl}t|dstj|_nddlm}| D]+\}}t |tj r|r|r|j du}|j g}|rB||j t|dd@tdd|j jDr^ WddS|j |j|j||j jd} | j |j |r{| j |j t||| Wdn1swYq"t |tjr|r|rt|j |j gdd+|j du}|j|jd|j|j jd } | j |j |r| j |j Wdn1swYt||| q"t ||j r|s|r|j du}tj |j|j||j jd} | j |j |r| j |j t||| q"t ||jrF|sF|rFtj|jd|j|j jd } | j |j | j |j t||| q"t||||d q"dS) zu Recursively converts the linear and layernorm layers of a model to their `transformers_engine` counterpart. zBUsing `convert_model` requires transformer_engine to be installed.rN LayerNorm) modifier_rankcss|] }|ddkVqdS)rN).0pr r e/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/accelerate/utils/transformer_engine.py 2sz convert_model..)bias params_dtype)epsr)to_transformer_engine_convert_linear _convert_ln)r ImportErrorrintel_transformer_enginehasattrnnrtransformer_engine.pytorchpytorchnamed_children isinstanceLinearrweightappendranyshape in_features out_featuresdtypecopy_setattrnormalized_shaper convert_model) modelrrrtenamemodulehas_biasparams_to_gather te_module new_moduler r r r(st       r(cCsbtstdtrddl}|j}nddlm}|j|j|jf}| D] }t ||r.dSq$dS)zS Returns whether a given model has some `transformer_engine` layer or not. zRUsing `has_transformer_engine_layers` requires transformer_engine to be installed.rNTF) rrrrrrrrTransformerLayermodulesr)r)r*module_cls_to_checkmr r r has_transformer_engine_layers_s   r5FcsJtstdtrddlmnddlmfdd}|_|S)z Wrapper for a model's forward method to apply FP8 autocast. Is context aware, meaning that by default it will disable FP8 autocast during eval mode, which is generally better for more accurate metrics. zLUsing `contextual_fp8_autocast` requires transformer_engine to be installed.r) fp8_autocastcsHp|j}|d|i|WdS1swYdS)N)enabled fp8_recipe)training)selfargskwargsr7r6r8 model_forwarduse_during_evalr r forwards  $z(contextual_fp8_autocast..forward)rrrrr6r __wrapped__)r>r8r?r@r r=r contextual_fp8_autocastvs rBc Cs.tstdtrddlm}d}d}nddlmm}ddlm}|j \}}|dur3| ni}d|vrCt |j |d|d<|dd}|dd}|rZ|sZtd ||rud |vrdtd d |vrltd |jdi|} n|jdi|} t|j| |} t|jdrt| ||_|S| |_|S)zC Applies FP8 context manager to the model's forward method zGUsing `apply_fp8_autowrap` requires transformer_engine to be installed.rNFz,MXFP8 block scaling is not available on HPU. fp8_formatuse_autocast_during_evaluse_mxfp8_block_scalingz&MXFP8 block scaling is not available: amax_compute_algoz=`amax_compute_algo` is not supported for MXFP8 block scaling.amax_history_lenz<`amax_history_len` is not supported for MXFP8 block scaling.__func__r )rrrintel_transformer_engine.reciperecipe transformer_engine.common.recipecommonrrfp8check_mxfp8_support to_kwargsgetattrFormatpop ValueErrorMXFP8BlockScalingDelayedScalingrBr@rr) r)fp8_recipe_handler te_recipeis_fp8_block_scaling_availablemessager*r<r?rEr8 new_forwardr r r apply_fp8_autowraps:      r[)TTT)F) typesrtorch.nnrimportsrr operationsrr(r5rBr[r r r r s    E