o i@sddlmZddlmZddlmZmZddlmZddl m Z m Z e e Zer4ddlZddlmZ     ddddZGdddeZGdddeZdS)) annotations)Any)is_torch_availablelogging) ConversionOps)get_module_from_nameshould_convert_moduleNcuda:0Fmodeltorch.nn.Modulemodules_to_not_convertlist[str] | None quant_config dict | None compute_dtype torch.dtypedevicestr pre_quantizedboolreturnc Csddlm}|dur g}t|D]D\}}t|tjsqt||s#q|d\} } } | r2| | n|} ||s:|j nd|s@|j nd|sH|j dund|||dd} t | | | q|S)a Replace nn.Linear modules with empty SINQLinear modules. Args: model: The model to modify modules_to_not_convert: List of module names to skip quant_config: SINQ quantization config dict (None for pre-quantized models) compute_dtype: Computation dtype for the quantized layers device: Device string for the quantized layers pre_quantized: Whether loading a pre-quantized checkpoint Returns: The modified model with SINQLinear modules r) SINQLinearN.FT) in_features out_featuresbiasrrruse_unpack_kernel)sinq.sinqlinear_hfrlist named_modules isinstancennLinearr rpartition get_submodulerrrsetattr)r r rrrrr full_namemodule parent_path_ child_nameparent sinq_layerr.`/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/integrations/sinq.pyreplace_with_sinq_linear s*      r0c@s*eZdZdZddZ   ddd dZdS) SinqQuantizea' Param-level ConversionOp for SINQ (from FP weights). At load time, for each `Linear.weight` that should be quantized: - The SINQLinear module already exists (created in _process_model_before_weight_loading) - We just call quantize() on it with the loaded weight tensor cC ||_dSN hf_quantizerselfr5r.r.r/__init__\ zSinqQuantize.__init__N input_dictdict[str, Any]r torch.nn.Module | Nonefull_layer_name str | Nonerdict[str, torch.Tensor]c Ks^tt|\}}t|tr|dn|}t||\} } | ||dur*||d| _iS)NrT) nextiteritemsr!rrquantizediscard_is_hf_initialized) r7r:r r= missing_keyskwargsr*values weight_tensorr( tensor_namer.r.r/convert_s  zSinqQuantize.convert)NNNr:r;r r<r=r>rr?__name__ __module__ __qualname____doc__r8rKr.r.r.r/r1Ssr1c@s(eZdZdZddZ  ddd dZdS)SinqDeserializea0 ConversionOp for loading *pre-quantized* SINQ checkpoints. Checkpoint layout (what `SINQLinear.state_dict` produces) is, per module: .W_q .bias .meta WeightConverter in the quantizer is configured so that: - we group ".W_q", ".meta", ".bias" as input_dict - conceptually treat them as belonging to ".weight" - and call this SinqDeserialize.convert to load the state into the existing SINQLinear. The returned dict is {} because we load directly into the module. cCr2r3r4r6r.r.r/r8r9zSinqDeserialize.__init__Nr:r;r r<r=r>rr?c Kst|D]\}}t|tr|d||<q|d}|d}|d} |dus-|durBtt|}t|tr>|d}||iSt||\} } ||d} | durV| | d<| | d| _ iS)Nrz.W_qz.metaz.bias)W_qmetarT) rrBr!getr@rArHrload_state_dictrE) r7r:r r=rGkvrSrTrr(r*stater.r.r/rKs*       zSinqDeserialize.convert)NNrLrMr.r.r.r/rRvs rR)NNNr F)r r r rrrrrrrrrrr ) __future__rtypingrtransformers.utilsrrcore_model_loadingrquantizers.quantizers_utilsrr get_loggerrNloggertorchtorch.nnr"r0r1rRr.r.r.r/s"      3#