o 0iEz@sdZddlZddlZddlmZddlmZmZddlm Z m Z ddl m Z ddl Z ddlmZmZd d lmZd d lmZmZd d lmZmZmZd d lmZer\ddlmmZereddl m!Z!ddZ"ddZ#ddZ$ddZ%ddZ&e"ddddZ'dnddZ(dd Z)d!d"Z*d#d$Z+d%d&Z,d'd(Z-d)d*Z.d+d,Z/d-d.Z0Gd/d0d0e1Z2d1d2Z3d3d4Z4e3d5d6Z5d7e fd8d9Z6d7e fd:d;Z7doddCe j?dDe j@dEe jAdFe jBdGe jCdHi ZDdIdJeDEDZFdKdLZGdqdMe jHfdNdOZIe3dodPeJfdQdRZKdodPeJfdSdTZLdrdUdVZMdodWdXZNGdYdZdZeOZPe4dsd[d\ZQdod]d^ZRe3dtdadbZSdcddZTGdedfdfZUdgdhZVdidjZWedudldmZXdS)vzB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)Mapping)contextmanager nullcontext)update_wrapperwraps)Any)AcceleratorState PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_npu_availableis_torch_distributed_availableis_torch_xla_available)is_torch_version)ReduceOpcCs t|tjSN) isinstancetorchTensortensorr]/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/accelerate/utils/operations.pyis_torch_tensor-s rc Cs2t|tjjtjjtjjtjjtjjtjjtjj Sr) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrrrris_torch_xpu_tensor1sr%cCs t|tSr)rr tensor_inforrris_tensor_information> r(cCst|tot|dot|dS)z Checks if `data` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. _asdict_fields)rtuplehasattrdatarrr is_namedtupleBsr0cCs$t|r t|t|St||S)zO Cast a generator to the same type as obj (list, tuple, or namedtuple) )r0typelist)obj generatorrrr honor_typeJs r5F test_typeerror_on_other_typecst|ttfrt|fdd|DSt|tr/t|fdd|DS|r>|gRiSrRtdt|djdjd|S) ad Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs (additional keyword arguments, *optional*): Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3s.|]}t|gRdVqdS)r6Nrecursively_apply).0oargsr8funckwargsr7rr ns z$recursively_apply..cs0i|]\}}|t|gRdqS)r6r9r;kvr=rr wsz%recursively_apply..zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr,r2r5rr1items TypeError__name__)r?r/r7r8r>r@rr=rr:Us, r:c s4t|s t|dr^dkrdz|jdWSty%|YStyF}ztr:ttr9dn|WYd}~nd}~wwz|jdWSty]|YSwt|tt frst |fdd|DSt|t rtt rgndurgt |fd d |DS|S) a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. tonpuznpu:0) non_blockingznpu:Nc3s |] }t|dVqdS)rK skip_keysNsend_to_device)r;tdevicerKrMrrrAsz!send_to_device..c s.i|]\}}||vr |nt|dqS)rLrN)r;rCrPrQrrrEsz"send_to_device..)rr-rIrGAssertionErrorrrintr,r2r5rstrr1rF)rrRrKrMerrorrrQrrOsF       rOcCdd}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. cSst|j|jdS)N)shapedtype)rrXrYrrrr_get_data_structuresz/get_data_structure.._get_data_structurer9)r/rZrrrget_data_structure r[cCrW)a: Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. cSs t|jSr)r2rXrrrr _get_shaper)zget_shape.._get_shaper9)r/r]rrr get_shaper\r^cCsdd}t||tdS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cSstj|jd|jiSNrY)remptyrXrYr&rrr_initialize_tensorz.initialize_tensors.._initialize_tensorr7)r:r()data_structurerarrrinitialize_tensorssrecCst|tttfrt|dkrtdt|dt|ttfr%t|dSt|tr9|D] }t||Snt|t j sIt dt|d|j dS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz&Cannot find the batch size from empty .z0Can only find the batch size of tensors but got ) rr,r2rlen ValueErrorr1find_batch_sizekeysrrrGrX)r/rCrrrris      ric Cs$zt|WSttfyYdSw)a Same as [`utils.operations.find_batch_size`] except will ignore if `ValueError` and `TypeErrors` are raised Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. N)rirhrGr.rrrignorant_find_batch_sizes  rkcCrW)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. cSs,|}|jtjkr|tj}|Sr)detachcpurYrbfloat16rIfloat32tolistrrrr_convert_to_list!s   z!listify.._convert_to_listr9)r/rqrrrlistifys rrcCs"dd}t||dd}t|S)NcSs0|jdkr |d}|s|}t|S)Nr)ndimclone is_contiguous contiguousxm all_gatherrrrr_tpu_gather_one.s   z$_tpu_gather.._tpu_gather_oneTr8)r:rw mark_step)rryresrrr _tpu_gather-s r}csJttjjjjdkrtddrtjfdd}t ||ddS)Nrz<=z2.8csjdkr dsjdur@jdkr@tjjj j d}||j dg ddRSfddt jD}tj|tj|ddS) NrgloorYrRr csg|]}tqSr)r empty_liker;_rrr \z8_gpu_gather.._gpu_gather_one..dim)rsrtrurvbackendrr` num_processesnumelrYrRviewsizerange distributedrxcat)routput_tensors gather_opstaterr_gpu_gather_oneDs    z$_gpu_gather.._gpu_gather_oneTrz) r rrall_gather_into_tensorrRr1rr synchronizer:)rrrrr _gpu_gather<s  rc@seZdZdZdS)DistributedOperationExceptionz An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rH __module__ __qualname____doc__rrrrrcsrctfdd}|S)zv Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. c stjtjks tjs|i|Sjdj}d|vr$|d}n|d}tjjt |jkrNt d|d|jjdtjjdtjjd|d t |}t |g}|ddur| |dt|k}|sd d d t|D}t d |d ||i|S)Nrfrrz%One or more of the tensors passed to z were not on the z+ while the `Accelerator` is configured for z. Please move it to the z before calling z - cSs g|] \}}d|d|qS)zProcess z: r)r;irXrrrrs z5verify_operation..wrapper..znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )r distributed_typer NOdebugrrHrRr1 find_devicerr^ gather_objectcountrgjoin enumerate)r>r@ operationrshapesoutputare_sameprocess_shape_strfunctionrrwrapperqs8   z!verify_operation..wrapperrrrrrrverify_operationlsrcr)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c sNz|i|WSty&}zjdj}td|d|d}~ww)NrfzError found while calling `z1`. Please see the earlier error for more details.)rrrH)r>r@errrrrs z"chained_operation..wrapperrrrrrchained_operations rcCs.tjtjkr t|Stjtvrt|S|S)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )r rr XLAr}r rrrrrgathers   robjectcCs2ddttjD}tj||dd|DS)NcSsg|]}dqSrrrrrrrsz&_gpu_gather_object..cSsg|] }|D]}|qqSrr)r;yxrrrrs)rr rrrall_gather_object)routput_objectsrrr_gpu_gather_objectsrcCs.tjtjkr tdtjtvrt|S|S)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supported)r rr rNotImplementedErrorr r)rrrrrs   rcCsddd}t||d|dS)NrcSstjj||d|S)Nsrc)rr broadcast)rrrrr_gpu_broadcast_onesz*_gpu_broadcast.._gpu_broadcast_oneT)r8rrr9)r/rrrrr_gpu_broadcasts rbroadcast tensorcsht|ttfrt|fddt|DSt|tr)t|fdd|DSt |fddS)Nc3s*|]\}}t|d|dVqdS)rnameN_tpu_broadcast)r;rrPrrrrA(z!_tpu_broadcast..cs(i|]\}}|t|d|dqS)rrrrBrrrrE(z"_tpu_broadcast..c|Srrrrrrz _tpu_broadcast..) rr2r,r5rrr1rFrw mesh_reduce)rrrr)rrrrs  r cCsi|]\}}||qSrrrBrrrrErrEcCsd}t}tj|tj|jd}|dur/|j}t|j}tjt ||gtd|dt |d<t |dd}|| }t|ddd }|dd}||fS) ze Grabs the shape of `tensor` only available on one process and returns a tensor of its shape irNrYr sum reductionrr) r rr`rTrRrXTENSOR_TYPE_TO_INTrYrr2rgreducenonzero)rmax_tensor_dimensionr base_tensorrX tensor_dtyperYrrrgather_tensor_shapes (   rreturncCs@t}t|\}}|durtj|t|d|j}t|ddS)a Copies a tensor that only exists on a single device and broadcasts it to other devices. Differs from `broadcast` as each worker doesn't need to know its shape when used (and tensor can be `None`) Args: tensor (`torch.tensor`): The tensor that should be sent to all devices. Must only have it be defined on a single device, the rest should be `None`. Nrrr)r rrzerosTENSOR_INT_TO_DTYPErIrRr)rrrXrYrrrcopy_tensor_to_devices s   r from_processcCs8tjtjkrt||ddStjtvrt||dS|S)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr)r rr rrr r)rrrrrrs   rcs^tjtjkrt|D]\}}td|fdd||<q |Stjtvr-tj j |d|S)a Broadcast a list of picklable objects from one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listcrrrrrrrr?rz'broadcast_object_list..r) r rr rrrwrr rrbroadcast_object_list) object_listrrr3rrrr0s  rcCsdd}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. cSs||Srr)r tensor_slicerrr _slice_tensorSz$slice_tensors.._slice_tensorr9)r/r process_indexrrrrr slice_tensorsEs rcstdttfrtdfddttdDStdtr9tdfdddDStdt j sLt dtdt j dS)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3s*|]tfddDdVqdS)cg|]}|qSrrr;drrrrgz)concatenate...rN concatenater;r/rrrrAgrzconcatenate..cs(i|]tfddDdqS)crrrrrCrrrirz*concatenate...rrrrrrrEirzconcatenate..z%Can only concatenate tensors but got r) rr,r2r5rrgrr1rjrrrGrrrrrrYs *(rc@s eZdZdS)CannotPadNestedTensorWarningN)rHrrrrrrrosrcCsddd}t||d|||dS) a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. rFc st|ddrtdt|St|jkst|j kr|Sdkr*t|j7tj|j|jdd}t | }t fdd|D|jkrO|S|jt }|<| t||}|rwtfddtt|D}ntfd dtt|D}|||<|S) N is_nestedFzHCannot pad nested tensors without more information. Leaving unprocessed.r)rRc3s|]}|VqdSrr)r;srrrrAszFpad_across_processes.._pad_across_processes..c3s2|]}|krtntdVqdSrslicer;rrmax_sizeold_sizerrrAs" c3.|]}|krtdntdVqdSrNrrrrrrrA,)getattrwarningswarnrrgrXrrrRrrmmaxr2 new_zerosr,r) rr pad_index pad_firstrsizesnew_size new_tensorindicesrrr_pad_across_processess4    z3pad_across_processes.._pad_across_processesT)r8rr r NrrFr9)rrr r rrrrpad_across_processesss " rcCsddd}t||d|||dS)z Takes a `tensor` of arbitrary size and pads it so that it can work given `num_processes` needed dimensions. New tensors are just the last input repeated. E.g.: Tensor: ([3,4,4]) Num processes: 4 Expected result shape: ([4,4,4]) rc s||}|||}||dkr||}n|||}|||@kr'dkr-nn||}|jt}|||d<|t|}tfddtt|D} ||| <|S)Nrr c3rrrrrrrrArz@pad_input_tensors.._pad_input_tensors..)rXr2r r,rrg) r batch_sizerr remainder last_inputsto_padr rrrrr_pad_input_tensorss      z-pad_input_tensors.._pad_input_tensorsT)r8rrrNrr9)rrrrrrrrpad_input_tensorss rmean?cCsddd}t||d||dS) aW Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valid on XLA. Returns: The same data structure as `data` with all the tensors reduced. rrcSst}|}|jtjkr|S|jtjkr'tttj |g|tn|jj t vr5t j |tj|dkr>||j}|S)Nr)r rtrr rrrwr{ all_reduce REDUCE_SUMvaluer rrrSUMr)rrscaler cloned_tensorrrr_reduce_across_processess     z(reduce.._reduce_across_processesT)r8rrNrrr9)rrrr!rrrrs  rcCsdd}dd}t|||dS)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. cSs|Sr)floatrrrr_convert_to_fp32 rz)convert_to_fp32.._convert_to_fp32cSs$t|s t|do|jtjtjfvSr_)rr-rYrfloat16rnrrrr_is_fp16_bf16_tensor sz-convert_to_fp32.._is_fp16_bf16_tensorrcr9)rr$r&rrrconvert_to_fp32s r'c@s(eZdZdZddZddZddZdS) ConvertOutputsToFp32ae Decorator to apply to a function outputting tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. cCs||_t||dSr) model_forwardr)selfr)rrr__init__"szConvertOutputsToFp32.__init__cOst|j|i|Sr)r'r))r*r>r@rrr__call__&rbzConvertOutputsToFp32.__call__cCs td)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)r*rrr __getstate__)sz!ConvertOutputsToFp32.__getstate__N)rHrrrr+r,r/rrrrr(s   r(cstfdd}|_|S)Ncs|i|Srr)r>r@r)rrforward2sz(convert_outputs_to_fp32..forward)r( __wrapped__)r)r1rr0rconvert_outputs_to_fp32/s r3cCs~t|tr|D]}t|}|dur|Sq dSt|ttfr4|D]}t|}|dur1|Sq#dSt|tjr=|jSdS)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvaluesrr,r2rrrR)r/r3rRrrrr;s"   rTccs|tjtjkstjdurtjst}nddl}|jj ||||d}| dVWddS1s7wYdS)z Wrapper around `deepspeed.runtime.zero.GatheredParameters`, but if Zero-3 is not enabled, will be a no-op context manager. Nr) modifier_rank fwd_moduleenabled) r rr DEEPSPEEDdeepspeed_pluginis_zero3_init_enabledr deepspeedzeroGatheredParameters)paramsr5r6r7gather_param_contextr;rrrr=Ps  "r=)FNr)rrr)NNrr")NNT)Yrr-rcollections.abcr contextlibrr functoolsrrtypingrrrr r constantsr dataclassesr rimportsrrrversionsrtorch_xla.core.xla_modelcore xla_modelrwtorch.distributedrrr%r(r0r5r:rOr[r^rerirkrrr}r Exceptionrrrrrrrrr#doublehalfrnuint8int8int16int32int64boolrrFrrrrrTrrrr UserWarningrrrrr'r(r3rr=rrrrs        34' #         7( )