o i:1@sdZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z mZe r:dd lmZd d lmZiZeed dGdddZGdddZedddZd1ddZd2d#d$Zd3d4d)d*ZeZd5d+d,Zd3d-d.d/d0ZdS)6z Contains the logic for automatic additional output capture with our forward decorators. This mostly describe the hooks used and the logic to make capture thread/context safe. ) annotationsN) ContextVar) dataclasswraps) TYPE_CHECKING)is_torchdynamo_compilingrequires)nnPreTrainedModel)torch)backendsc@s>eZdZUdZded<dZded<dZded <dZded <dS) OutputRecordera Configuration for recording outputs from a model via hooks. Attributes: target_class (Type): The class (e.g., nn.Module) to which the hook will be attached. index (Optional[int]): If the output is a tuple/list, optionally record only at a specific index. layer_name (Optional[str]): Name of the submodule to target (if needed), e.g., "transformer.layer.3.attn". class_name (Optional[str]): Name of the class to which the hook will be attached. Could be the suffix of class name in some cases. ztype[nn.Module] target_classrintindexN str | None layer_name class_name)__name__ __module__ __qualname____doc____annotations__rrrrre/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/utils/output_capturing.pyr's   rc@s0eZdZdZddZddZddZdd Zd S) CompileableContextVara Convenience wrapper around a ContextVar for usage with `torch.compile`. This behaves exactly as a `ContextVar`, except when compilation is triggered in which case it behaves as a simple global variable. This is useful as `torch.compile` cannot trace the `get` method of `ContextVar`. This however means that the access to the underlying variable is not thread-safe when compilation is triggered. cCst||d|_||_d|_dS)NdefaultF)r context_var global_var compiling)selfnamer!rrr__init__Bs zCompileableContextVar.__init__cCs(|jr|jStrd|_|jS|jSNT)r$r#r is_compilingr"get)r%rrrr*Gs  zCompileableContextVar.getcCs"tr ||_d|_dS|j|Sr()r r#r$r"set)r%valuerrrr+Ss  zCompileableContextVar.setcCs&|jr d|_d|_dS|j|dS)NF)r$r#r"reset)r%tokenrrrr-[s zCompileableContextVar.resetN)rrrrr'r*r+r-rrrrr:s  routput_collectorr module nn.ModulekeystrrrreturnNonecsfdd}||dS)zaInstall the forward hook needed to capture the output described by `key` and `index` in `module`.cst}|dus|vrdSdkr%t|dkr%||dt|ts3||dS|durD||dSdS)N hidden_statesr)_active_collectorr*keyslenappend isinstancetuple)r0argsoutputcollected_outputsrr2rroutput_capturing_hookjs  z;install_output_capuring_hook..output_capturing_hookN)register_forward_hook)r0r2rrArr@rinstall_output_capuring_hookgsrC parent_module module_name capture_tasks list[tuple[str, OutputRecorder]]cCsddlm}|D] \}}t||st||d||q t||d|dq |D],\}}|jdur\}}t|ts|g}|D]/}t|tsFd|vr+dnd}t|ts4dn|}t|ts=|nd} t| ||d}|||fqq|durU|nd}t |||d|_ dS)z Install the output recording hooks on all the modules in `model`. Tis will take care of correctly dispatching the `_can_record_outputs` property of each individual submodels in case of composite models. r6rrN)rrrT) _CAN_RECORD_REGISTRYr*r3 __class__itemsr;listrr:rL!_output_capturing_hooks_installed) rPrI capture_flagsrFr2 layer_specsrOrrrrrrrMs     rMcCs`t|ddrdStt|ddr WddSt|WddS1s)wYdS)z Check if the model already has output capturing hooks installed, and install them if it is not already the case. Note that this is thread-safe, in case 2 (or more) threads want to install them concurrently. rVFN)getattr_hook_installation_lockrM)rPrrrmaybe_install_capturing_hookss   "r[Ttie_last_hidden_statescs fdd}|dur||S|S)a Decorator to intercept specific layer outputs through hooks. The hooks are installed only once and lazily, the first time output capture is requested with the `output_xxx` kwargs/config. The implementation is fully context/thread safe, except when using `torch.compile`, as dynamo is unable to trace through `ContextVar` methods. Args: tie_last_hidden_states (`bool`, *optional*, defaults to `True`): Whether to overwrite `out.hidden_states[-1]` with the `out.last_hidden_state`. This is true for all language models and should be toggled off only if `out.hidden_states[-1]` has to be the hidden state before last layer norm, which is needed for some vision models (e.g. CLIP, SigLIP) cstfdd}|S)Nc sdtjdd}ttjpi}fdd|D}d|vr0dtjdd|d<d |vrAdtjdd|d <d d|D}t|d krTt t |}zg|Ri}Wt |nt |w|D]} | d krs}n/t |dr|| dd|| <|| |jnt |dr|| dd|| <|| |jt|| || <qt| dkrt|| trt|| dkrt|| d dd|| <t|| ddd|d| <qtt|| || <qtt|| || <qt|dur|}|S)N return_dictTc s6i|]}d|d|tjd|dqS)output_F)r*rYconfig).0kkwargsr%rr s(zHcapture_outputs..wrapped_fn..wrapper..cross_attentionsoutput_attentionsFoutput_cross_attentionsmask_decoder_attentionsoutput_mask_decoder_attentionscSs"i|] \}}|r|ddgqS)r_rQ)replace)rarbvrrrres"rr6vision_hidden_stateslast_hidden_state attentionsr rcross_)poprYr`rRr*r3rSrTr9r[r7r+r-hasattrr:rmror<r;rUto_tuple) r%r=rdr^capturable_flagsrecordable_keysr? output_tokenoutputsr2)funcr]rcrwrappersP      z4capture_outputs..wrapped_fn..wrapperr)ryrzr\)ryr wrapped_fns?z#capture_outputs..wrapped_fnNr)ryr]r{rr\rcapture_outputss Cr|)r0r1r2r3rrr4r5)rDr1rEr3rFrGr4r5)N)rPrrIrr4r5)rPrr4r5)r __future__r threading contextvarsr dataclassesr functoolsrtypingr import_utilsr r rr rJrrRrrr7rCrLrMLockrZr[r|rrrrs.        *