o
0i6 @ sj d dl Z d dlmZ d dlmZ d dlZd dlZd dlm Z ddl
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z eddrVd d l m Z nd d l!m Z e rhd dl"m# m$Z% dd
l&m'Z' dd l(m)Z) e'e*Z+
d!de,de-e. de-de-de-de/de/dee de0de0fddZ1 d"ddZ2d#d e/de0fd d Z3d$d e/fd d Z4dS )% N)Path)Optional)
load_model )
MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAMESAFE_MODEL_NAMESAFE_WEIGHTS_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAME WEIGHTS_NAMEget_pretty_nameis_cuda_availableis_hpu_availableis_mlu_availableis_musa_availableis_sdaa_availableis_torch_versionis_torch_xla_availableis_xpu_availableloadsavez>=z2.4.0)
GradScaler)
get_logger) PartialStateFT
output_dir model_states
optimizers
schedulers dataloaders
process_indexstepscalersave_on_each_nodesafe_serializationc
! C sL t | } t|D ]-\}
} | stnt} |
dkr | dd|
d} | | }
t| |
|| d td|
qt|D ].\}
}| } |
dkrKt
dnt
d|
d}| |}t| ||dd td| q:t|D ].\}
}| } |
dkr~t dnt d|
d}| |}t| ||dd td | qmt|D ]e\}
}|
dkrt dnt d|
d}| |}d
d l
m}m} t|j|r| }t||rt|||dd t|d dr|
dkrd
nd|
d}| |}| } t| | td|
d| q|dur"| } | t} t| | td| i } t d| d} || d< t | d< tj | d< t | d< t rNtj | d< t rZtj | d< nt! rftj" | d< n t# rqtj$ | d< t% r|tj& | d < t' rtj( | d < t) rt* | d < | | } t| | td | | S ) aw
Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory.
If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native
`pickle`.
Args:
output_dir (`str` or `os.PathLike`):
The name of the folder to save all relevant weights and states.
model_states (`List[torch.nn.Module]`):
A list of model states
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances to save their sampler states
process_index (`int`):
The current process index in the Accelerator state
step (`int`):
The current step in the internal step tracker
scaler (`torch.amp.GradScaler`, *optional*):
An optional gradient scaler instance to save;
save_on_each_node (`bool`, *optional*):
Whether to save on every node, or only the main node.
safe_serialization (`bool`, *optional*, defaults to `True`):
Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
r ._)r% r& zModel weights saved in .binFzOptimizer state saved in zScheduler state saved in r IterableDatasetShardSeedableRandomSampleruse_stateful_dataloaderdl_state_dict.bindl_state_dict_z Sampler state for dataloader z
saved in Nz Gradient scaler state saved in .pklr# random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_mlu_manual_seedtorch_sdaa_manual_seedtorch_musa_manual_seedtorch_hpu_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in )+r enumerater r
replacejoinpathr loggerinfo
state_dictr r
r data_loaderr+ r,
isinstancedataset get_samplergetattrtorchr r randomgetstatenp get_state
get_rng_stater xpuget_rng_state_allr mlur sdaar musar hpur cudar xm)!r r r r r! r" r# r$ r% r& istate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file
dataloader sampler_nameoutput_sampler_filer+ r, samplerdataloader_state_dict_name!output_dataloader_state_dict_filer@ output_scaler_filestates states_nameoutput_states_file rh Z/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/accelerate/checkpointing.pysave_accelerator_state> s~ ,
"
"
"
rj c # K s t }
|dvr td|du rd}n|dkrt j}|du r i }t| } t|D ]D\} } | dkr5d| nd}
| t |
d }| rSt | |fd
t
|i| q(| t |
d }t ||d }| j
|fi | q(td
t|D ]+\} }| dkrt d nt d| d }| |}t |fd|i|}||
| qvtd t|D ]'\} }| dkrt d nt d| d }| |}t |fi |}|
| qtd t|D ]e\} }| dkrt d nt d| d }| |}ddlm}m} t|j|r| } t| | r|t |} t|ddrA| dkr$dnd| d } | | } | rAt | fi |}|
| qtd |dur_| t} t | }!|
|! td z|t | t d| d}"d|"v rx|"d |
d< t |"d t j |"d t!"|"d t# rt!j$%|"d t& rt!j'%|"d n"t( rt!j)%|"d nt* rt!j+%|"d! nt!j,%|"d" t- rt."|"d# td$ W |
S t/y td% Y |
S w )&a
Loads states of the models, optimizers, scaler, and RNG generators from a given directory.
Args:
input_dir (`str` or `os.PathLike`):
The name of the folder to load all relevant weights and states.
models (`List[torch.nn.Module]`):
A list of model instances
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
process_index (`int`):
The current process index in the Accelerator state
scaler (`torch.amp.GradScaler`, *optional*):
An optional *GradScaler* instance to load
map_location (`str`, *optional*):
What device to load the optimizer state onto. Should be one of either "cpu" or "on_device".
load_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the `load` function.
load_model_func_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the model's `load_state_dict` method.
Returns:
`dict`: Contains the `Accelerator` attributes to override while loading the state.
)Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`Nrk rl r r( z .safetensorsdevicer) ) map_locationz%All model weights loaded successfullyro z(All optimizer states loaded successfullyz(All scheduler states loaded successfullyr r* r- Fr. r/ z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr0 r# r1 r2 r3 r4 r5 r6 r7 r9 r: z%All random states loaded successfullyz Could not load random states)0dict TypeErrorr rn r r; r= r existsr strr r load_state_dictr> r? r r
r rA r+ r, rB rC rD set_samplerrE r r rG setstaterI set_staterF
set_rng_stater rL set_rng_state_allr rN r rO r rP rR r rS Exception)# input_dirmodelsr r r! r" r$ ro load_kwargsload_model_func_kwargsoverride_attributesrT modelendinginput_model_filer@ rX rY input_optimizer_fileoptimizer_stater[ r\ input_scheduler_filescheduler_stater^ r_ input_sampler_filer+ r, ra rb input_dataloader_state_dict_fileinput_scaler_file scaler_statere rh rh ri load_accelerator_state s '
"
"
"
r indexc C sD t |d| d }tdt| d| t| ||d dS )zL
Saves the state of `obj` to `{path}/custom_checkpoint_{index}.pkl`
custom_checkpoint_r0 zSaving the state of z to )r% N)r r> r? r r r@ )objpathr r%
save_locationrh rh ri save_custom_state: s r c C sB | d| d}t dt| d| | t|ddd dS ) z
Loads the state of `obj` at `{path}/custom_checkpoint_{index}.pkl`. Will always set `weights_only=False` when
loading the state.
z/custom_checkpoint_r0 zLoading the state of z from rk F)ro weights_onlyN)r> r? r rt r )r r r
load_locationrh rh ri load_custom_stateD s r )NFT)NNN)r F)r )5rG pathlibr typingr numpyrI rF safetensors.torchr utilsr r r r r
r r r
r r r r r r r r r r r r torch.ampr torch.cuda.amptorch_xla.core.xla_modelcore xla_modelrS loggingr rU r __name__r> rs listrp intboolrj r r r rh rh rh ri s\ X
}