o i< @sjddlmZddlmZmZddlmZmZmZddl m Z ddl Z ddl m Z ddlmZddlmZerHdd lmZdd lmZdd lmZd gZd ede dfddZd ede dfddZd'ddZd(ddZd)d edeedBdefddZ d*dee j!dee"e j#Bd ede$e$e j!fd!d"Z%ed#ed$Z& d*d%e&dee"e j#Bd ede$e&fd&d Z'dS)+) OrderedDict)IteratorSequence)cast TYPE_CHECKINGTypeVar)TypeIsN_get_device_index)Module)comm) ScriptMethod) ScriptModule) EnabledProxy replicatemodulereturnrcCddl}t||jjSNr) torch.jit isinstancejitrrtorchr]/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/torch/nn/parallel/replicate.py_is_script_modulerr cCrr)rr_Cr rrrr_is_script_methodrrcCsddl}|jSr)rrrrrrr_init_script_module! r!rcCsddl}|jjjSr)torch.jit._stater_state_enabledr rrr_is_jit_enabled'r"r&memocCsdtdttfdd}tsdS|durt}||t|r2|||tdd||DS|D]}||vr=q6t ||sEdSq6dS) NrrcSs|}t||SN)modulesnext)rgenrrrdescendant_modules6sz0_replicatable_module..descendant_modulesTcss|]}t|VqdSr()r).0 descendantrrr Ds z'_replicatable_module..F) r rr&setaddrupdateallchildren_replicatable_module)rr'r,childrrrr54s$   r5Ftensorsdevicesdetachcsbddlm}|rt|Stdkr/|j|gRfddtdttDSgS)Nr) Broadcastcs g|] }||tqSr)len)r-i tensor_copiesr7rr `sz0_broadcast_coalesced_reshape..)torch.nn.parallel._functionsr:r broadcast_coalescedr;applyrange)r7r8r9r:rr=r_broadcast_coalesced_reshapeSs    rDT)boundnetworkcst|std|s gSdd|D}t|}t|}ddt|D}t|||}t|}g}g} |D]} | jrE|sE| | q8| | q8ddt|D} ddt| D} t|||d} t| |d d}t| }d d|Di}t|D]\}}|||<t |D]}| }t |_| |qq~t|D]\}}|jD]6\}}|durt |D] }||}d|j|<qq||}t |D]}||}t||||qq|jD]?\}}|durt |D] }||}d|j|<qq||}t |D]}||}|||}t|||||j|<q q|jD]K\}} | durIt |D]}||}d|j|<q8q+| jrW|sW| }| | }n|}| | }t |D]}||}t|||||qaq+qfd dt |DS) NzJCannot replicate network where python modules are children of ScriptModulecSsg|]}t|dqS)Tr )r-xrrrr?xzreplicate..cSi|]\}}||qSrr)r-idxparamrrr |rIzreplicate..cSrJrrr-rKbufrrrrMrIcSrJrrrNrrrrMrI)r9TcSsg|]}gqSrr)r-_rrrr?scsg|] }tt|dqS)r)rrE)r-j module_copiesrrr?s)r5 RuntimeErrorr;list parameters enumeraterDbuffers requires_gradappendr)rC_replicate_for_data_parallelr_former_parameters_modulesitemssetattr _parameters_buffers)rGr8r9 num_replicasparams param_indices param_copiesrX buffers_rgbuffers_not_rgrObuffer_indices_rgbuffer_indices_not_rgbuffer_copies_rgbuffer_copies_not_rgr)module_indicesr<rrQreplicakeyr6 module_idxrL param_idx param_copy buffer_copies buffer_idxrrRrrks                            )rr)rrr()F)( collectionsrcollections.abcrrtypingrrrtyping_extensionsrr torch._utilsr torch.nn.modulesr torch.nn.parallelr torch._Cr rrr#r__all__robjectrr!r&r0boolr5TensorintdevicerUrDrErrrrrsN           "