o i@sddlmZddlmZddlmZmZer!ddlZddlm Z e e Z GdddeZ Gdd d ejjZGd d d e jZdd eedBfddZdS)) ConversionOps)should_convert_module)is_torch_availableloggingNc @sJeZdZddZ d deeeejfdedBdeeejffddZ dS) EetqQuantizecCs ||_dSN) hf_quantizer)selfr r `/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/integrations/eetq.py__init__s zEetqQuantize.__init__N input_dictfull_layer_namereturnc Ksnt|d\}}|d}|j}t|}t|tj d\}}| |}| |}|||d|iS)NrF_scales) tupleitemsdevicetorcht contiguouscpueetq_kernels_hub quant_weightsint8to) r rrkwargs_value value_device int8_weightscalesr r r converts  zEetqQuantize.convertr) __name__ __module__ __qualname__r dictstrlistrTensorr#r r r r rs rc@s&eZdZedddZeddZdS)EetqLinearMMFunctionNcCs:|||||t|||}|dur||}|S|}|Sr)save_for_backwardr w8_a16_gemm)ctxxweightr"biasoutputr r r forward0s zEetqLinearMMFunction.forwardcCsp|j\}}}}t|jd|j|j}t|||}|j dr2| d | dd d}|dddfS)Nr) saved_tensorsreyeshaperrdtyperr-needs_input_gradsqueezematmul transpose unsqueeze)r. grad_outputinputr0r"r1identity grad_inputr r r backward8s    zEetqLinearMMFunction.backwardr)r$r%r& staticmethodr3rBr r r r r+/s  r+cs,eZdZejdffdd ZddZZS) EetqLinearFcsjttjtj||f|ddd|_ttj|tjd|_|r0ttj|tjd|_ dSd|_ dS)N)r8F) requires_grad) superr nn Parameterremptyr0float16 weight_scalesr1)r in_features out_featuresr8r1 __class__r r r Hs  zEetqLinear.__init__cCst||j|j|j}|Sr)r+applyr0rKr1)r r?r2r r r r3QszEetqLinear.forward)r$r%r&rrr r3 __classcell__r r rNr rDGs rDFmodules_to_not_convertc Csddlm}|dad}|rinddi}|D]=\}}t||s"qtd&t|tj rFt |j |j fd|j dui|}|||d }Wdn1sPwYq|s]td |S) a A helper function to replace all `torch.nn.Linear` modules by `EetqLinear` modules. Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`list[`str`]`, *optional*, defaults to `None`): Names of the modules to not convert in `EetqLinear`. In practice we keep the `lm_head` in full precision for numerical stability reasons. r4) get_kernelz#kernels-community/quantization-eetqFr8Nmetar1TzYou are loading your model using eetq but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.) hub_kernelsrSr named_modulesrrr isinstancerGLinearrDrLrMr1 set_submoduleloggerwarning) modelrR pre_quantizedrShas_been_replaced module_kwargs module_namemodule new_moduler r r replace_with_eetq_linearVs4     rc)NF)core_model_loadingrquantizers.quantizers_utilsrutilsrrrtorch.nnrG get_loggerr$rZrautogradFunctionr+ModulerDr)r(rcr r r r s