o
ij @ s
d Z ddlZddlZddlZddlmZ ddlZddlmZ ddl m
Z
m Z m Z ddl
mZ ddlmZ eeZeG d d
d
ZG d d d ZeG d
d deZG dd dZG dd deZG dd deZG dd deZG dd deZG dd deeZ dS )zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)tqdm )IntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc @ s~ e Zd ZU dZdZedB ed< dZeed< dZ eed< dZ
eed< dZ eed < dZ eed
< dZ
edB ed < dZeed < dZeed
< dZeed< dZeeeef ed< dZedB ed< dZedB ed< dZedB ed< dZeed< dZeed< dZeed< dZ edB ed< dZ eeeeB eB eB f dB ed< dZ ed dB ed< d d Z d efd d Z e!d efd!d"Z"d#d$ Z#d%d& Z$dS )' TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`list[dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_global_step (`int`, *optional*):
When tracking the best model, the step at which the best metric was encountered.
Used for setting `best_model_checkpoint`.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`list[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevant callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_global_stepbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc C s | j d u rg | _ | jd u ri | _d S t| jtrd S i }| jD ]6}t|ts/tdt| |jj}||v rOt|| t sE|| g||< ||
| q | ||< q || _d S )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr callbackname r- _/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/trainer_callback.py
__post_init__t s&
zTrainerState.__post_init__ json_pathc C sX t jt| dddd }t|ddd}|| W d dS 1 s%w Y dS ) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r* r0 json_stringfr- r- r. save_to_json s "zTrainerState.save_to_jsonc C sH t |dd }| }W d n1 sw Y | di t|S )z3Create an instance from the content of `json_path`.r6 r7 Nr- )r= readr9 loads)clsr0 r@ textr- r- r. load_from_json s
zTrainerState.load_from_jsonc C sN dD ]"}t || d}|dur$|dk rt|| }t| | d| qdS )z
Calculates and stores the absolute value for logging,
eval, and save steps based on if it was a proportion
or not.
)r evalsave_stepsNr )getattrmathceilsetattr)r* argsr
step_kind num_stepsr- r- r.
compute_steps s zTrainerState.compute_stepsc C sj |j dur|jdur| |j| _d| _|dur#ddlm} ||| _|| _|| _| | _| | _ dS )zI
Stores the initial training references needed in `self`
Nr ) hp_params)
hp_name_trialr r transformers.integrationsrR r
r r r )r* trainerr
r trialrR r- r- r. init_training_references s
z%TrainerState.init_training_references)%r&
__module__ __qualname____doc__r float__annotations__r intr
r r r r r r r r r' r! strr r r r boolr r r r r r/ rA classmethodrF rQ rX r- r- r- r. r
" s8
; $
r
c @ s* e Zd ZdZdefddZedd ZdS )r" aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc C s t d)NzInternal class that just calls the list of callbacks in order.c C sj g | _ |D ]}| | q|| _|| _|| _|| _d | _d | _tdd | j D s3t
d| j d S d S )Nc s s | ]}t |tV qd S N)r DefaultFlowCallback.0cbr- r- r. s z+CallbackHandler.__init__..zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r* r r r r r r r- r- r. __init__ s zCallbackHandler.__init__c C sh t |tr| n|}t |tr|n|j}|dd | jD v r,td| dd | j | j| d S )Nc S s g | ]}|j qS r- )r% )r cr- r- r.
s z0CallbackHandler.add_callback..zYou are adding a zH to the callbacks of this Trainer, but there is already one. The currentzlist of callbacks is
:)r r$ r% r r r r r( )r* r+ r cb_classr- r- r. r s
z CallbackHandler.add_callbackc C sd t |tr | jD ]}t ||r| j| | S qd S | jD ]}||kr/| j| | S q d S r r r$ r remover* r+ r r- r- r. pop_callback s
z CallbackHandler.pop_callbackc C sF t |tr| jD ]}t ||r| j| d S qd S | j| d S r r r r- r- r. remove_callback s
z CallbackHandler.remove_callbackc C s d dd | jD S )Nr4 c s s | ]}|j jV qd S r )r% r& r r- r- r. r s z0CallbackHandler.callback_list..)joinr rd r- r- r. r s z CallbackHandler.callback_listrN r) rx c C | d|||S )Nr|
call_eventr* rN r) rx r- r- r. r| zCallbackHandler.on_init_endc C d|_ | d|||S )NFr~ )rl r r r- r- r. r~ z CallbackHandler.on_train_beginc C r )Nr r r r- r- r. r r z CallbackHandler.on_train_endc C r )NFr )rm r r r- r- r. r r z CallbackHandler.on_epoch_beginc C r )Nr r r r- r- r. r r z CallbackHandler.on_epoch_endc C s" d|_ d|_d|_| d|||S )NFr )rp ro rn r r r- r- r. r s z CallbackHandler.on_step_beginc C r )Nr r r r- r- r. r r z%CallbackHandler.on_pre_optimizer_stepc C r )Nr r r r- r- r. r r z!CallbackHandler.on_optimizer_stepc C r )Nr r r r- r- r. r r z CallbackHandler.on_substep_endc C r )Nr r r r- r- r. r r zCallbackHandler.on_step_endc C d|_ | jd||||dS )NFr r )ro r r* rN r) rx r r- r- r. r
zCallbackHandler.on_evaluatec C s | j d||||dS )Nr r r r r- r- r. r s zCallbackHandler.on_predictc C r )NFr )rn r r r- r- r. r r zCallbackHandler.on_savec C r )NFr )logs)rp r )r* rN r) rx r r- r- r. r r zCallbackHandler.on_logc C r )Nr r r r- r- r. r r z"CallbackHandler.on_prediction_stepc K s | j d|||fi |S )Nr r rz r- r- r. r s z CallbackHandler.on_push_beginc
K sP | j D ]"}t|||||f| j| j| j| j| j| jd|}|d ur%|}q|S )N)r r r r r r )r rJ r r r r r r )r* eventrN r) rx r{ r+ resultr- r- r. r s&
zCallbackHandler.call_eventN) r& rY rZ r[ r r r r propertyr r r
rk r| r~ r r r r r r r r r r r r r r r r- r- r- r. r s2
r c @ s< e Zd ZdZdededefddZdededefddZd S )
r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rN r) rx c K s |j dkr |jr d|_|jtjkr |j |j dkr d|_|jtjkr3|j |j dkr3|j |j kr3d|_
|j t jkrI|j
dkrI|j |j
dkrId|_|j |jkr[d|_|j t jkr[d|_|S )Nr Tr )r logging_first_steprp logging_strategyr STEPSr
eval_strategyr
eval_delayro
save_strategyr r rn r
rl rz r- r- r. r 8 s"
z DefaultFlowCallback.on_step_endc K sF |j tjkr d|_|jtjkr|j|jkrd|_|jt jkr!d|_
|S )NT) r r EPOCHrp r r r ro r r rn rz r- r- r. r X s z DefaultFlowCallback.on_epoch_endN) r& rY rZ r[ r r
rk r r r- r- r- r. r 3 s r c @ s\ e Zd ZdZddefddZdd Zdd Zdd d Zd
d Z dd Z
dddZ dd Z d
S )ProgressCallbackz
A [`TrainerCallback`] that displays the progress of training or evaluation.
You can modify `max_str_len` to control how long strings are truncated when logging.
d max_str_lenc C s d| _ d| _|| _dS )a!
Initialize the callback with optional max_str_len parameter to control string truncation length.
Args:
max_str_len (`int`):
Maximum length of strings to display in logs.
Longer strings will be truncated with a message.
N) training_barprediction_barr )r* r r- r- r. r n s
zProgressCallback.__init__c K s |j r t|jdd| _d| _d S )NT)total
dynamic_ncolsr )r r r
r current_steprz r- r- r. r~ { s
z ProgressCallback.on_train_beginc K s* |j r| j|j| j |j| _d S d S r )r r updater r rz r- r- r. r s z ProgressCallback.on_step_endNc K sJ |j r!t|r#| jd u rtt|| jd u dd| _| jd d S d S d S )NT)r leaver r )r r r r lenr r )r* rN r) rx r r{ r- r- r. r s
z#ProgressCallback.on_prediction_stepc K ( |j r| jd ur
| j d | _d S d S r r r closerz r- r- r. r
z ProgressCallback.on_evaluatec K r r r rz r- r- r. r r zProgressCallback.on_predictc
K s |j rM| jd urOi }| D ].\}}t|tr,t|| jkr,dt| d| j d||< t|tr8|d||< q|||< q|dd } | j t| d S d S d S )Nz%[String too long to display, length: z > z/. Consider increasing `max_str_len` if needed.].4gr )
r r rf r r_ r r r\ popr> )
r* rN r) rx r r{ shallow_logsrh ri _r- r- r. r s
zProgressCallback.on_logc K s |j r
| j d | _d S d S r )r r r rz r- r- r. r s
z ProgressCallback.on_train_end)r r )
r& rY rZ r[ r^ r r~ r r r r r r r- r- r- r. r h s
r c @ s e Zd ZdZdddZdS )PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc K s< | dd }|jr |d urdd | D }t| d S d S )Nr c S s( i | ]\}}|t |tr|d n|qS )r )r r\ )r rh ri r- r- r.
s ( z*PrinterCallback.on_log..)r r rf print)r* rN r) rx r r{ r r- r- r. r s zPrinterCallback.on_logr )r& rY rZ r[ r r- r- r- r. r s r c @ sN e Zd ZdZddededB fddZd d
Zd d Zd
d Z de
fddZ dS )EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
r early_stopping_patienceearly_stopping_thresholdNc C s || _ || _d| _d S )Nr r r early_stopping_patience_counter)r* r r r- r- r. r s
z EarlyStoppingCallback.__init__c C sX |j rtjntj}|jd u s |||jr#t||j | jkr#d| _d S | jd7 _d S )Nr r )greater_is_betternpgreaterlessr absr r )r* rN r) rx metric_valueoperatorr- r- r. check_metric_value s
z(EarlyStoppingCallback.check_metric_valuec K s: |j std |jd usJ d|jtjksJ dd S )NzUsing EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.zBEarlyStoppingCallback requires metric_for_best_model to be definedzAEarlyStoppingCallback requires IntervalStrategy of steps or epoch)load_best_model_at_endr r metric_for_best_modelr r NOrz r- r- r. r~ s z$EarlyStoppingCallback.on_train_beginc K sl |j }|ds
d| }||}|d u r!td| d d S | |||| | j| jkr4d|_d S d S )Neval_z@early stopping required metric_for_best_model, but did not find z so early stopping is disabledT) r
startswithgetr r r r r rl )r* rN r) rx r r{ metric_to_checkr r- r- r. r s
z!EarlyStoppingCallback.on_evaluaterb c C s | j | jdd| jidS )N)r r r rw r rd r- r- r. r) s zEarlyStoppingCallback.state)r r ) r& rY rZ r[ r^ r\ r r r~ r r! r) r- r- r- r. r s
r ) r[ r; r9 rK r numpyr tqdm.autor
trainer_utilsr r r
training_argsr utilsr
get_loggerr& r r
r" rk r r r r r r r- r- r- r. s2
,= 5J