o i @sddlZddlmZmZddlmZeeZeZ dej dej j dej fdd Z    ddej j dej d ej d ej d ej dBdededBdedBdedBdedBdeej dffddZdS)N)_flash_attention_forward!flash_attn_supports_top_left_mask)loggingquerymodulereturncCsR|jtjkr'tdrtdSt|jdr|jjStdd|Dj jSdS)ziIf the query is in float32, return a target dtype compatible with flash attention. Return None otherwise.cuda _is_quantizedcss"|] }t|tjjr|VqdS)N) isinstancetorchnnLinear).0layerrk/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/integrations/flash_attention.py s z#get_target_dtype..N) dtyper float32is_autocast_enabledget_autocast_dtypehasattrconfignextmodulesweight)rrrrrget_target_dtype s    rkeyvalueattention_maskdropoutscalingsliding_windowsoftcap is_causalc Ks| ddr td|jd} tdd|jDrtd|dd}|dd}|dd}t||} | dur;| n|j} t ||||f| | ||||t | |j j t |d rW|jndd | } | dfS) Noutput_attentionsFzFlash Attention does not support `output_attentions=True`. Please set your attention to `eager` if you want any of these features.rcss|]}|dkVqdS)rNr)rdimrrrr/sz*flash_attention_forward..zTensor query has shape with a zero dimension. FlashAttention does not support inputs with dim=0. Please check your input shapes or use SDPA instead. layer_idx) query_lengthr&r" softmax_scaler$r%use_top_left_mask target_dtypeattn_implementationr*)getlogger warning_onceshapeany ValueError transposerr&r_use_top_left_maskr_attn_implementationrr*)rrrr r!r"r#r$r%r&kwargsseq_lenr. attn_outputrrrflash_attention_forwardsD      r<)rNNNN)r modeling_flash_attention_utilsrrutilsr get_logger__name__r1r7Tensorr Modulerrfloatintbooltupler<rrrrsD