o ieB @sddlmZddlmZddlmZddlmZddlm Z e r%ddl Z e e Z  d'd e jd e jd e jdBd ed e jf ddZde jjde jde jde jd e jf ddZ d(d e jd e jde jdBd e jfddZ   d)d e jd e jd e jdBde jdBd ed e jf ddZde jjde jde jde jd e jf ddZGdddeZeZde jd e jfd d!Z d(ddd"d#ee jjdBd ed$ed ee jjfd%d&ZdS)*)Callable)wraps)logging)GeneralInterface)is_torch_availableNFinputweightbias is_transposedreturncCsJ|rt|d|d}n t||dd}|dur#||}|S)aBatched linear layer supporting optional bias and transposed weights. Args: input (`torch.Tensor`): Input tensor of shape (batch_size, input_dim). weight (`torch.Tensor`): Weight tensor of shape (batch_size, output_dim, input_dim) if transposed is `False`, else of shape (batch_size, input_dim, output_dim). bias (`torch.Tensor`, *optional*): Bias tensor of shape (batch_size, output_dim). Default is `None`. is_transposed (`bool`, *optional*, defaults to `False`): Whether the weight tensor is transposed. Returns: `torch.Tensor`: Output tensor of shape (batch_size, output_dim). N)torchbmm unsqueezesqueeze)rr r r outr_/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/integrations/moe.py_batched_linearDs rself hidden_states top_k_index top_k_weightscCsF|j}|d}|d}|d}tj||ddd|d}|d} |d} | |jk} | d|jd} ||} |j | }|j | }|j rQ|j | nd}|j r[|j | nd}t| |||jd}||}t||||jd}| j| jkr| d| } || d}|| d|j}||||jdd}||jS)Nrrdevicer r dim)rsizerarangerexpandreshape num_expertsclamp gate_up_proj down_projhas_biasgate_up_proj_biasdown_proj_biasrr _apply_gateshapegathertodtypeviewsum)rrrrr num_top_k num_tokens hidden_dim token_idxsample_weights expert_ids valid_maskexpert_ids_clampedselected_hidden_statesselected_gate_up selected_downselected_gate_up_biasselected_down_bias gate_up_out gated_outout_per_samplefinal_hidden_statesrrrbatched_mm_experts_forwardfs6   "           rCoffscCsTttjjdrtjjj||j||dSttdr&tj||j||dStd)aGrouped matrix multiplication dispatcher that uses torch.nn.functional.grouped_mm if available, else falls back to torch._grouped_mm. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, output_dim, input_dim). offs (`torch.Tensor`, *optional*): Offsets tensor indicating the boundaries of each group in the input tensor. Returns: `torch.Tensor`: Output tensor of shape (S, output_dim). Raises: ImportError: If neither `torch.nn.functional.grouped_mm` nor `torch._grouped_mm` is available, indicating that the PyTorch version is incompatible. grouped_mmrD _grouped_mmzNeither torch.nn.functional.grouped_mm nor torch._grouped_mm is available. Please make sure you are using a PyTorch version that includes grouped_mm (2.9+).) hasattrrnn functionalrEr.r/rG ImportError)rr rDrrrrGs rGcCs>|r t|||d}n t||dd|d}|dur||}|S)a*Grouped linear layer supporting optional bias and transposed weights. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, output_dim, input_dim) if transposed is `False`, else of shape (num_experts, input_dim, output_dim). bias (`torch.Tensor`, *optional*): Bias tensor of shape (num_experts, output_dim). Default is `None`. offs (`torch.Tensor`, *optional*): Offsets tensor indicating the boundaries of each group in the input tensor. is_transposed (`bool`, *optional*, defaults to `False`): Whether the weight tensor is transposed. Returns: `torch.Tensor`: Output tensor of shape (S, output_dim). rFrN)rG transpose)rr r rDr rrrr_grouped_linears rNcCsr|j}|d}|d}|d}tj||ddd|d}|d} |d} ||} t| } t| } | | }| | }| | }|j}|j }|j rU|j |nd}|j r_|j |nd}|j dkrj|n|}tj||jd|jdd}tj|dtjd}t|||||jd}||}t|||||jd}||d}|| }||||jdd }||jS) Nrrrr cpu)binsminmax)rr/rr)rr rr!rr"r#argsortr&r'r(r)r*typefloatinthistcr$cumsumint32rNr r+r0r1r.r/)rrrrrr2r3r4r5r6r7r:perminv_perm expert_ids_gsample_weights_gselected_hidden_states_gr;r<r=r> histc_inputnum_tokens_per_expertoffsetsr?r@out_per_sample_grArBrrrgrouped_mm_experts_forwards>   "        rccs8eZdZdZeedZdededeffdd Z Z S)ExpertsInterfacez9Interface for registering custom experts implementations.) batched_mmrEexperts_implementationdefaultr csB|dur tdn|dkr||vrtd|dt||S)zfReturn the requested `experts_implementation`. Also strictly check its validity, and raise if invalid.Na You tried to access the `ExpertsInterface` with a `config._experts_implementation` set to `None`. This is expected if you use an Expert Module as a standalone Module. If this is not the case, something went wrong with the dispatch of `config._experts_implementation`eager`zL` is not a valid experts implementation registered in the `ExpertsInterface`)logger warning_onceKeyErrorsuperget)rrfrg __class__rr get_interface4s zExpertsInterface.get_interface) __name__ __module__ __qualname____doc__rCrc_global_mappingstrrrq __classcell__rrrorrd,s "rdr?cCs |jddd\}}|||S)a Default gating mechanism: splits the gate_up_out into gate and up parts, applies the activation function to the gate part, and multiplies it with the up part. Args: gate_up_out (`torch.Tensor`): The output tensor from the gate and up projection of shape (S, 2 * intermediate_dim). Returns: `torch.Tensor`: The gated output tensor of shape (S, intermediate_dim). rrr)chunkact_fn)rr?gateuprrr_default_apply_gateFs r})r r( experts_classr(cs<dttjjdttjjffdd }|dur||S|S)aVDecorator to modify experts class to support different experts implementations. Args: experts_class (`type[torch.nn.Module]`, *optional*): The experts class to modify. If not provided, returns a decorator that can be applied to the class. is_transposed (`bool`, *optional*, defaults to `False`): Whether the expert weights are stored in transposed format. has_bias (`bool`, *optional*, defaults to `False`): Whether the expert layers include bias terms. Returns: `type[torch.nn.Module]`: The modified experts class. r~r csX|j|jtfdd}tfdd}t|ds$t|_||_||_|S)Ncs.||g|Ri|||_|_|_dSN)configr(r )rrargskwargs)r(r original_initrr__init__is z=use_experts_implementation..wrapper..__init__cs&t|jj}||g|Ri|Sr)ALL_EXPERTS_FUNCTIONSrqr_experts_implementation)rrrexperts_forward)original_forwardrrforwardpsz.wrapper..forwardr+)rrrrHr}r+)r~rrr(r )rrrwrapperes z+use_experts_implementation..wrapperN)rTrrIModule)r~r r(rrrruse_experts_implementationTs(r)NFr)NNF)collections.abcr functoolsrutilsr utils.genericrutils.import_utilsrr get_loggerrrrjTensorboolrrIrrCrGrNrcrdrr}rTrrrrrs      - " < $ & F