o i %@sddlmZddlmZddlmZmZddlmZddl m Z ddl m Z er-dd l Z er5dd lmZeeZGd d d e Zd S) ) annotations) TYPE_CHECKING)is_torch_availablelogging) SinqConfig) HfQuantizer)get_module_from_nameN)PreTrainedModelcseZdZUdZdZded<d,fdd Zd-d d Zed-d d Z ddZ d.ddZ d/ddZ d0ddZ d1dd Zd!d"Zd#d$Z %d2d3d(d)Zd4d*d+ZZS)5SinqHfQuantizera HF v5 quantizer for SINQ. Modes: - method="sinq" (default): * weight-only SINQ * param-level ConversionOps (`SinqQuantize`) during load for pure language models (each Linear.weight is turned into a SINQLinear module) * module-level quantization after load for multimodal models - method="asinq": * A-SINQ (activation-aware) SINQ quantization Tbool requires_parameters_quantizationquantization_configrc s$tj|fi|d|_d|_dS)NF)super__init___normalized_device_str_do_param_level_sinq)selfrkwargs __class__h/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/quantizers/quantizer_sinq.pyr1s zSinqHfQuantizer.__init__returncCdSNTrrrrris_serializable7szSinqHfQuantizer.is_serializablecCrrrrrrr is_trainable:szSinqHfQuantizer.is_trainablecCs@|durtjrdtji}nddi}td|d|S)Ncpuz:The device_map was not initialized. Setting device_map to zJ. If you want to use the model for inference, please set device_map='auto')torchcuda is_availablecurrent_deviceloggerinfo)r device_maprrrupdate_device_map>s z!SinqHfQuantizer.update_device_mapdtype torch.dtypecCs|durtj}||_|SN)r"bfloat16r*)rr*rrr update_dtypeKszSinqHfQuantizer.update_dtypeNonecOsddlm}|s tdtjstd|d}t |t r7t | }t |dkr7tdt|d|jjd krD|jsFtd dSdS) Nr)is_sinq_availablezMThe 'sinq' package is not installed. Please install it with: pip install sinqzNo CUDA device is available. Quantization and inference will run on the CPU. Please note that this will significantly slow down inference speed and increase quantization time.r(rzkSinqHfQuantizer: multi-GPU device_map detected, but SINQ currently supports only a single CUDA device. Got z. Please use device_map=None.asinqzYou are using `method='asinq'` in the quantization config. Right now the calibrated version of SINQ is not supported in Hugging Face, please refer and use the official SINQ repository `to quantize a model with this method. )utilsr0 ImportErrorr"r#r$r&warningget isinstancedictsetvalueslen RuntimeErrorsortedrmethod pre_quantized ValueError)rargsrr0r(device_map_valuesrrrvalidate_environmentQs*      z$SinqHfQuantizer.validate_environmentcfgr7c CsJddlm}|j}|t|j|jdurt|jndddddt|j|dS)zI Build the dict that SINQLinear expects as quant_config. r)sinq_base_quant_configNFr)nbits group_size quant_zero quant_scale view_as_floataxis tiling_moder=)sinq.sinqlinear_hfrDr=intrErFstrrK)rrCsinq_base_quant_config_fnr=rrr_build_sinq_quant_dictms z&SinqHfQuantizer._build_sinq_quant_dictmodelr param_namerNc Ksnddlm}|jr dS|jjdkrdS|jsdSt||\}}|dkr%dSt||}t|dd}|o4| } | S)a- Called per-parameter to decide whether to run `SinqQuantize` on it. - If `self.pre_quantized`, we do *not* quantize again (handled by SinqDeserialize instead). - For method="asinq": return False (ASINQ is not supported in Hugging Face). - For method="sinq": True only for SINQLinear.weight not in modules_to_not_convert. Note: After _process_model_before_weight_loading(), the modules are already SINQLinear, not nn.Linear. We check for SINQLinear modules that are not yet quantized (ready=False). r) SINQLinearFr1weightreadyT) rLrSr>rr=rr r6getattr) rrQrRrrSmodule tensor_nameis_sinqis_readyresultrrrparam_needs_quantizations     z(SinqHfQuantizer.param_needs_quantizationcCsddlm}||S)z Return the ConversionOps used for param-level quantization (Sinq). The actual SINQLinear construction is in integrations/sinq.py. r) SinqQuantize)integrations.sinqr])rr]rrrget_quantize_opss z SinqHfQuantizer.get_quantize_opscCs>ddlm}|jrddlm}|gddg||gdgSgS)a4 If `pre_quantized=True`, interpret a checkpoint produced by SINQLinear.state_dict: .W_q .bias .meta via a WeightConverter + SinqDeserialize so that we reconstruct a SINQLinear module instead of a plain nn.Linear. r)WeightConverter)SinqDeserialize)z.W_qz.metaz.biasz.weight)source_patternstarget_patterns operations)core_model_loadingr`r>r^ra)rr`rarrrget_weight_conversionss   z&SinqHfQuantizer.get_weight_conversionsNkeep_in_fp32_moduleslist[str] | Nonec Ksddlm}|||jjpg||_|jjdko|j |_|jr"dn||j}t |t rFt t | d}t |trAd|}nt|}n tjrMdnd}|||j||j||jd }dS) a Called on meta-initialized model, before loading any weights. For SINQ, we replace nn.Linear modules with empty SINQLinear modules here. The actual quantization happens later in SinqQuantize.convert() when weights are loaded. r)replace_with_sinq_linearsinqNrzcuda:zcuda:0r!)modules_to_not_convert quant_config compute_dtypedevicer>)r^riget_modules_to_not_convertrrkr=r>rrPr6r7nextiterr9rMrNr"r#r$r*) rrQr(rgrrisinq_quant_dict first_device device_strrrr$_process_model_before_weight_loadings(      z4SinqHfQuantizer._process_model_before_weight_loadingcKsddlm}||S)aq Called after *all* weights have been loaded. For SINQ: 1. Move non-SINQLinear modules to GPU (embeddings, norms, lm_head, etc.) - SINQLinear modules already have GemLite buffers on GPU - We skip moving SINQLinear's W_q/meta to avoid memory duplication 2. Patch HF save/load methods for SINQ serialization r)patch_hf_pretrained_io) sinq.hf_iorv)rrQrrvrrr#_process_model_after_weight_loadings z3SinqHfQuantizer._process_model_after_weight_loading)rr)rr )r*r+rr+)rr/)rCrrr7)rQr rRrNrr r,)rQr rgrh)rQr )__name__ __module__ __qualname____doc__r__annotations__rrpropertyrr)r.rBrPr\r_rfrurx __classcell__rrrrr !s"       " ! +r ) __future__rtypingrr2rrutils.quantization_configrbaser quantizers_utilsr r"modeling_utilsr get_loggerryr&r rrrrs