o i&c@sVddlmZmZerddlZddlmZddlmZddlmZddl m Z m Z e e ZgdZed d ZGd d d eZGd ddeZGdddeZddZddZejdddejdedejfddZejdddejdedejfddZGdddejZd d!Zd"d#Zd$d%Z d&d'Z!d(d)Z"d*d+Z#d/d,e$e%dBfd-d.Z&dS)0)is_torch_availableloggingN)nn)contextmanager) ConversionOps)get_module_from_nameshould_convert_module)gg?g?g?g@g@g@g@ggggggggccstreddl}t||jr|j}n t|tr||}t|dd}|dkrA|j| dV WddS1s.topkr)binsmaxiT)r)src_indxdst_indx)osr7routing GatherIndx RoutingData ScatterIndxcompute_expt_data_torchrrr distributedget_world_sizerenvirongetrzsoftmaxsortgatherr}histcrr_rywherer)logits n_expts_actrrrrr world_sizerank replace_valuen_tokens n_expts_totn_local_expertslocal_expert_startlocal_expert_end n_gates_padr expt_scal expt_indx sort_indiceshistvar topk_indx gate_indxrrr expt_data hit_expertsrrrrouting_torch_distksN       4rc Csddlm}|r|rt|drt}ntjj}|jd}| d|j j }t j ||j j|j j}t|j|||j j\}}}Wdn1sMwY|j||||d} | |d|j j } | |fS)Nr _is_hookedr')r)torch.distributedr is_availableis_initializedrrr7rrzr}router hidden_dimr functionallinearweightbiasrrtop_kexperts) rrdistr batch_size router_logitsrrr routed_outrrr mlp_forwards   rc Ksddlm}|d}|d}|d} |d} |d} |d} d D]W} | |vr}| dur;|||||| | | | }| d }| d }t||d d d |t||r}t||r}tt||t||}t|| tj | |t ||t ||q&dS)Nrshard_and_distribute_moduler# empty_param casting_dtype to_contiguousr device_mesh)r)r*rPrQ.rZ) integrations.tensor_parallelrrr>rsplitrrrr rrr_delattr)rD param_name param_value target_device dq_param_namerArr#rrrrr rF blocks_attr scales_attrrVrrr dequantizes<          rcCst||}tj|Sr)rr rr)rTrUrVrrrrSs  rSc Ks|jj|jj|jj}}}ddlm} |d} |d} |d} |d} |d}|d}d |vrB|d d d d }d|vrR|d d dd }|durb| | || || | ||nt|| d ddt j j |dd|d }|d}t ||}t ||}|jjdkr=|jjdkr?|d }|dkr|||jdd }n ||d |jd}t |d|dkrtt drt jjnd}||}||}t|t|dd |dd |\}}Wdn1swY|dkrt ||j|jdg|_n t ||j|jg|_t|||t||d||||ddt||t||~dSdSdS)q This transforms the weights obtained using `convert_gpt_oss.py` to load them into `Mxfp4GptOssExperts`. rrr#rrrrr rTr r'rPrrUrQNrZFrmetar)r r acceleratorr r(r+r,r.)r8r9r:r;r rrsplitr>r r rrrrr sizer}rrrcurrent_acceleratorr_rrr<r6Sizerrzr )rDrrrr7rAr9r:r;rr#rrrrr rFrrrTrU local_expertsrEr/rrrload_and_swizzle_mxfp4sf       $           rc Csx|jj|jj|jj}}}|d} t|d|dkr(ttdr&tj j nd}| | }| | }|dkrE| | |jdd}n | | d|jd}t|d|dkrYd}t|t|d d|d d|\} } Wd n1sxwY|dkrt| |j|jdg| _n t| |j|jg| _||jvr|j|=t||| t||d || ||d d d S)rrr rrr r)rr'r(Nr+r,r.)r8r9r:r;rrrr rrr r_rr}rrr<r6rrrzr=r>) rTrUrDrFrr7r9r:r;rrEr/rrrrX,s<     rXmodules_to_not_convertc Cs|jr|Sddlm}|dad}|D]H\}}t||sq|jjdkrH|jsHt d| |t |j d}Wdn1sCwY|jjd kr]|js]d d l m}|t||_q|setd |S) aD Public method that replaces the expert layers of the given model with mxfp4 quantized layers. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`Mxfp4Config`, defaults to `None`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*, defaults to `None`): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. rZ) get_kernelz(kernels-community/gpt-oss-triton-kernelsF GptOssExpertsrTN GptOssMLPr) MethodTypezYou are loading your model using mixed-precision FP4 quantization but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)r hub_kernelsrr7 named_modulesr rrJr r set_submoduler4rtypesr"rrloggerwarning)r#quantization_configrrhas_been_replaced module_namerDr"rrrreplace_with_mxfp4_linear[s,      r,)NN)'utilsrrr r contextlibrcore_model_loadingrquantizers.quantizers_utilsrr get_loggerrJr'r{rrrOrWr5r<r`rfrrrrrNr4rrrrSrrXr3rr,rrrrsT     0, : CD!C/