o i*@s"ddlZddlZddlmZddlmZddlZejjj ejjj ejjj ejjj ejjj ejjjejjjejjjejjjejjjejjjejjjejjjejjjdZ dGdejded ed ejdBd ejf d d Z dGdejdeded ejdBd ejf ddZ dejded ejfddZ dejd ejfddZ dejd ejfddZ dejd ejfddZdHdejded ejfddZdIdejded ejdBd ejfd d!ZdIdejded ejdBd ejfd"d#Z $ % dJdejded&ed'ed ejdBd ejf d(d)Z $ % dJdejded&ed'ed ejdBd ejf d*d+Z   , - dKdejdededed ed ejdBd ejfd.d/Z  dLdejded ejdBd ejfd0d1Z dMdejd3eded ejdBd ejf d4d5Zdejd6ejd ejfd7d8ZdNd:d;Zdd?Z d@Z!edAdBZ"edCdDZ#edEdFZ$dS)ON) defaultdict)contextmanager)uniform_normal_ constant_ones_zeros_eye_dirac_xavier_uniform_xavier_normal_kaiming_uniform_kaiming_normal_ trunc_normal_ orthogonal_sparse_?tensorab generatorreturncC$t|ddstd||||dS|S)N_is_hf_initializedFr)rrrgetattrTORCH_INIT_FUNCTIONS)rrrrr]/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/initialization.pyr* rmeanstdcCr)NrFr)r!r"rr)rr!r"rrrrr2r rvalcC t|ddstd||dS|S)NrFr)r#r)rr#rrrr: rcCt|dds td|S|S)NrFrrrrrrr@  rcCr&)NrFrrr'rrrrFr(rcCr&)NrFr rr'rrrr Lr(r groupscCr$)NrFr )r*r)rr*rrrr Rr%r gaincC"t|ddstd|||dS|S)NrFr r+rrrr+rrrrr X r cCr,)NrFr r-rr.rrrr ^r/r fan_in leaky_relumode nonlinearitycC&t|ddstd|||||dS|S)NrFr rr2r3rrrrr2r3rrrrr d  r cCr4)NrFrr5rr6rrrrrr7r@cCs(t|ddstd||||||dS|S)NrFr)r!r"rrrr)rr!r"rrrrrrrs rcCr,)NrFrr-rr.rrrrs r{Gz?sparsitycCr)NrFr)r;r"rr)rr;r"rrrrrr rothercCsBt|ddst ||WdS1swY|S)NrF)rtorchno_gradcopy_)rr<rrrr?s   r?normalcCstjj|\}}|dkr|}n|dkr|}n |dkr!||d}d|}|dkr6t|t|dddS|d krEt|t|ddS|d krYtd |}t|| |dSt d |) Nr0fan_outfan_avgrtruncated_normalg۶%?)r"r@uniformzinvalid distribution ) r=nninit_calculate_fan_in_and_fan_outrmathsqrtrr ValueError)rr2 distributionr0rAdenomvarianceboundrrr_variance_scalings  rQcCt|dds t|ddd|S)NrFr0rDr2rMrrQr'rrr lecun_normal_ rUcCrR)NrFr0r@rSrTr'rrrdefault_flax_embed_init_rVrW) z torch.nn.initztorch.nn.modules.activationztorch.nn.modules.transformerztorch.nn.modules.linearztorch.nn.modules.lossztorch.nn.modules.batchnormztorch.nn.modules.convztorch.nn.modules.normalizationztorch.nn.modules.rnnztorch.nn.modules.sparsec cstt}zKtD]*}|tjvr2tj|}tD]}t||r1t|||||<t ||t |qqdVW| D]\}}| D] \}}t |||qCq;dS| D]\}}| D] \}}t |||q]qUw)a Guard the `torch.nn.init` primitive functions to behave exactly like the functions in this file, i.e. be protected against the `_is_hf_initialized` flag to avoid re-init if the param was already loaded. Usually, all models are using the init from `transformers` which are already guarded, but just to make extra sure and for remote code, we also use this context manager. N) rdictTORCH_MODULES_TO_PATCHsysmodulesrkeyshasattrrsetattrglobalsitems) originals module_namemodule func_name functionsfuncrrrguard_torch_init_functionss*     rgc csddlm}dd}tt}zQtD]'}|tjvr9tj|}tD]}t ||r8t |||||<t |||q"q|j }||_ dVW| D]\}}| D] \}}t |||qPqH||_ dS| D]\}}| D] \}}t |||qmqe||_ w)ac Disable weight initialization both at the torch-level, and at the transformers-level (`init_weights`). This is used to speed-up initializing an empty model with deepspeed, as we do not initialize the model on meta device with deepspeed, but we still don't need to run expensive weight initializations as we are loading params afterwards. r)PreTrainedModelc_dSNrargskwargsrrr empty_funcz#no_init_weights..empty_funcN)modeling_utilsrirrXrYrZr[rr\r]rr^ init_weightsr`) rirorarbrcrdoriginal_init_weightsrerfrrrno_init_weightss4       rtccs>ddlm}dd}z|j}||_dVW||_dS||_w)a Disable weight tying during loading with `from_pretrained`. This is needed as we want to have access to ALL weights in the state_dict during `from_pretrained`, and otherwise tying them would remove them from it, as it's called in `post_init` when instantiating. r)rhc_rjrkrrlrrrro$rpz"no_tie_weights..empty_funcN)rqri tie_weights)rirooriginal_tie_weightsrrrno_tie_weightss rw)rrN)r))rN)rr0r1N)rrr8r9N)r)N)r:N)r0r@)%rJrZ collectionsr contextlibrr=rGrHrrrrrr r r r r rrrrrTensorfloat Generatorintstrr?rQrUrWrYrgrtrwrrrrs      &&          $