o
0i @ s d Z ddlmZ ddlmZmZmZ ddlZddlm Z m
Z
er)e r)ddl m Z dej
jfd d
Zd ed ee d
efddZd ed
efddZe
dej
jfddZe
defdej
jded dee fddZdS )z,
Needed utilities for torchao FP8 training.
)partial)
TYPE_CHECKINGCallableOptionalN )is_torchao_availabletorchao_required)Float8LinearConfigmodelc C s@ d\}}| D ]\}}t|tjjr|du r|}|}q||fS )z
Finds the first and last linear layer names in a model.
This is needed during FP8 to avoid issues with instability by keeping the first and last layers unquantized.
Ref: https://x.com/xariusrke/status/1826669142604141052
)NNN)
named_modules
isinstancetorchnnLinear)r
first_linear last_linearnamemodule r U/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/accelerate/utils/ao.py find_first_last_linear_layers s r fqnlayers_to_filterreturnc C s> t | tjjr| jd dks| jd dkrdS ||v r dS dS )a
A function which will check if `module` is:
- a `torch.nn.Linear` layer
- has in_features and out_features divisible by 16
- is not part of `layers_to_filter`
Args:
module (`torch.nn.Module`):
The module to check.
fqn (`str`):
The fully qualified name of the layer.
layers_to_filter (`List[str]`):
The list of layers to filter.
r FT)r r
r r in_features out_features)r r r r r r filter_linear_layers1 s r c C s t | \}}t| |||gdS )a
A filter function which will filter out all linear layers except the first and last.
For stability reasons, we skip the first and last linear layers Otherwise can lead to the model not training or
converging properly
Args:
module (`torch.nn.Module`):
The module to check.
fqn (`str`):
The fully qualified name of the layer.
r )r r )r r r r r r r #filter_first_and_last_linear_layersH s r c C s2 ddl m} | D ] \}}t||r dS q
dS )Nr ) Float8LinearTF) torchao.float8.float8_linearr r r )r
r r r r r r
has_ao_layers] s
r" configr module_filter_funcc C sB ddl m} t| \}}|du rtt||gd}|| ||d dS )a
Converts all `nn.Linear` layers in the model (except the first and last) to torchao's `Float8Linear` layer inplace.
Args:
model (`torch.nn.Module`):
The model to convert.
config (`torchao.float8.Float8LinearConfig`, *optional*):
The configuration for the FP8 training. Recommended to utilize
`torchao.float8.recipe_name_to_linear_config` to generate this. In general, the default config should be
sufficient (what is passed when set to `None`).
module_filter_func (`Callable`, *optional*, defaults to `filter_linear_layers`):
Optional function that must take in a module and layer name, and returns a boolean indicating whether the
module should be converted to FP8. Defaults to `filter_linear_layers`. See it for an example.
Example:
```python
from accelerate.utils.ao import convert_model_to_fp8_ao
model = MyModel()
model.to("cuda")
convert_to_float8_training(model)
model.train()
```
r )convert_to_float8_trainingNr )module_filter_fnr# )torchao.float8r% r r r )r
r# r$ r% r r r r r convert_model_to_fp8_aog s
r( )__doc__ functoolsr typingr r r r
importsr r r! r r Moduler strlistboolr r r" r( r r r r s.