o iM6@sddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZddlmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlm?Z?ddl@mAZAddlBmCZCddlDmEZEddlFmGZGide#de'de)d e5d!ed"e=d#e?d$e3d%e-d&e7d'e9d(e+d)e/d*eEd+e%d,eGd-eCe1e!e;eAd.ZHidede de d%e d ed!ed"ed#ed$ed'ed(e d)e d&ed*ed+e d,ed-eeeeed.ZIeJeKZLGd/d0d0ZMGd1d2d2ZNd3eOfd4d5ZPd6eOfd7d8ZQd9d:ZRdS);N) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8Config FPQuantConfig GPTQConfig HiggsConfig HqqConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SinqConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FPQuantHfQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SinqHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerawqbitsandbytes_4bitbitsandbytes_8bitgptqaqlmquantoquarkfp_quanteetqhiggshqqzcompressed-tensors fbgemm_fp8torchaobitnetvptqspqr)fp8z auto-roundmxfp4sinqc@s.eZdZdZedefddZeddZdS)AutoQuantizationConfigz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictcCs|d}|dds|ddr!|ddrdnd}tj|}n|dur)td|tvr;td|d ttt|}||S) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsrFrGsuffix target_clsrZ^/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/quantizers/auto.pyrV|s    z AutoQuantizationConfig.from_dictcKsVtj|fi|}t|dddurtd|d|j}||}|jdi||S)Nquantization_configz)Did not found a `quantization_config` in z2. Make sure that the model is correctly quantized.rZ)rfrom_pretrainedgetattrrQr\rVupdate)rWpretrained_model_name_or_pathkwargs model_configrFr\rZrZr[r]s  z&AutoQuantizationConfig.from_pretrainedN)__name__ __module__ __qualname____doc__ classmethoddictrVr]rZrZrZr[rEvs rEc@s\eZdZdZedeeBfddZeddZedeeBdedBfd d Z e d d Z dS) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. r\cKsxt|tr t|}|j}|tjkr|jr|d7}n|d7}|tvr0t d|dt t t|}||fi|S)NrKrJrMrN) isinstancerhrErVrGrrPrHrTrQrSrU)rWr\rarGrYrZrZr[ from_configs     zAutoHfQuantizer.from_configcKstj|fi|}||S)N)rEr]rk)rWr`rar\rZrZr[r]s zAutoHfQuantizer.from_pretrainedquantization_config_from_argsNc Cs|durd}nd}t|trt|trt|}nt|}|dur9|jj|jjkr9td|jjd|jjdt|tt tt t t t frh|durh|}|D] \}}t|||qQ|dt|d7}|dkrzt|t t fszt||St||S) z handles situations where both quantization_config from args and quantization_config from model config are present. NzYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)rjrhrrVrE __class__rcrQrrr r rr get_loading_attributesitemssetattrrSrUwarningswarnloggerinfo)rWr\rl warning_msgloading_attr_dictattrvalrZrZr[merge_quantization_configssF        z*AutoHfQuantizer.merge_quantization_configscCs|dd}|dds|ddr"|ddrdnd}tj|}n|dur*td|tvr@td|d tt d dSd S) NrGrHFrIrJrKrLrMrNz~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) rOrrPrQrRrtwarningrSrTrU)rFrGrXrZrZr[supports_quant_methods   z%AutoHfQuantizer.supports_quant_method) rcrdrerfrgrrhrkr]rz staticmethodr|rZrZrZr[ris  <rimethodcfdd}|S)z-Register a custom quantization configuration.c6tvr tddt|tstd|t<|S)NzConfig '' already registeredz*Config must extend QuantizationConfigMixin)rRrQ issubclassr TypeErrorrWr~rZr[register_config_fn  z8register_quantization_config..register_config_fnrZ)r~rrZrr[register_quantization_config  rnamecr)zRegister a custom quantizer.cr)Nz Quantizer 'rz!Quantizer must extend HfQuantizer)rTrQrrrrrrZr[register_quantizer_fn(rz1register_quantizer..register_quantizer_fnrZ)rrrZrr[register_quantizer%rrcCst|d}|rt|jsd}|s|dur,|r t|j||_n||_tj|j|d}nd}|dur[|j||d||}||}| |}t |jdds[|jj }t |d||d<|||fS)Nr\F) pre_quantized) device_map weights_only dequantizevaluequant) hasattrrir|r\rzrkvalidate_environmentupdate_device_mapupdate_tp_planupdate_ep_planr^rG)configr\rr user_agentr hf_quantizerrGrZrZr[get_hf_quantizer5s4      r)Srrmodels.auto.configuration_autorutilsrutils.quantization_configrrrrr r r r r rrrrrrrrrrrrrbaserquantizer_aqlmrquantizer_auto_roundr quantizer_awqrquantizer_bitnetr quantizer_bnb_4bitr!quantizer_bnb_8bitr"quantizer_compressed_tensorsr#quantizer_eetqr$quantizer_fbgemm_fp8r%quantizer_finegrained_fp8r&quantizer_fp_quantr'quantizer_gptqr(quantizer_higgsr) quantizer_hqqr*quantizer_mxfp4r+quantizer_quantor,quantizer_quarkr-quantizer_sinqr.quantizer_spqrr/quantizer_torchaor0quantizer_vptqr1rTrR get_loggerrcrtrEristrrrrrZrZrZr[s  `                                 )v