o i @sddlZddlZddlmZddlmZmZmZddlm Z m Z e e Z e r,ddlZer4ddlmZddZ    d%d ed d ed d edBdedBdedeff ddZ    d%d ed d ed d edBdedBdedeff ddZ   d&d d d ed d edBdedBdedeff ddZ   d&d d d ed d edBdedBdedeff ddZ   d&d d d ed d edBdedBdedeff ddZeeeeedZGdddeddZGd d!d!Zd'd ed"edBfd#d$ZdS)(Nwraps) TYPE_CHECKINGOptional TypedDict)is_torch_availablelogging)PreTrainedConfigcs2ddddddtdfdd }|S) ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. Nc Sst|d}|dur|j}|j}d}|jjd}n|j|}t||d}|d}|jj|d}||krgt||dsQt|} | |j||d|d\} } |j |d | d d t ||d | dS| |}|j |d |d d t ||d |dS)zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.rN original_max_position_embeddings_original_inv_freq__long_inv_freqseq_len layer_typeinv_freqF persistent long_inv_freqoriginal_inv_freq) torchmax rope_typerconfigrope_parametersgetattrhasattrROPE_INIT_FUNCTIONSregister_buffersetattrto) self position_idsdevicerrrrprefixr rope_init_fnrrr(b/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_update.s4     z6dynamic_rope_update..longrope_frequency_updatec Sst|d}|dur|j}|j}|j}d}n|j|}t||d|j}t||d}|d}||kr[t|} | |j|||d\} |_|j |d| d d t ||d|||j kr||j kr| |}|j |d|d d t ||d |t ||d|j dSdSdS) a dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rNr _max_seq_len_cachedr rrrFrr) rrrmax_seq_len_cachedrrrrattention_scalingr r!original_max_seq_lenr") r#r$r%rrrr,rr&r'rr(r(r)dynamic_frequency_updateQs4    z5dynamic_rope_update..dynamic_frequency_updatecs|dur|jn|j|}|durd|ini}d|vr'||fd|ji|n|dkr7||fd|ji||||fi|S)Nrdynamicr%longrope)rr%)r#xr$rrkwargsr/r* rope_forwardr(r)wrapperwsz$dynamic_rope_update..wrapperNr)r5r6r(r4r)dynamic_rope_update!s #& r8rr r%z torch.devicerrreturnz torch.Tensorc Cs||dur |j|n|j}|d}|d}|dd}t|ddp)|j|j}t||} d} d|tjd| dtj d j |tj d | } | |} | | fS) a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). Nfactor rope_thetapartial_rotary_factor?head_dimrdtyper%rA) standardize_rope_paramsrgetr hidden_sizenum_attention_headsintrarangeint64r"float) rr%rrrope_parameters_dictr:baser<r>dimattention_factorrr(r(r)'_compute_linear_scaling_rope_parameterss!  ,rOc Cs||dur |j|n|j}|d}|dd}t|d|j|j}t||}|d} d} |dur8|j}nt|t j rNt |t j |j|j |jd}nt||j}|| ||j| d||d }d|t jd |d t jd j|t jd |} | | fS) a Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The default sequence length used to update the dynamic RoPE at inference time * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which `factor` will be accessed. The value of `factor` is used to determine the new base frequency, along with the current sequence length (seq_len), the maximum positional embeddings (max_position_embeddings), and the computed dimensionality (dim) of the rotary embeddings. If seq_len <= max_position_embeddings, this factor has no effect. If seq_len <= max_position_embeddings, this factor effectively stretches the context window using an exponent derived from `dim`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. If `None` or shorter than max_position_embeddings, this value will be overridden by max_position_embeddings. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). Nr;r<r=r>r:rAr%rr?rr@rB)rCrrDrrErFrGmax_position_embeddings isinstancerTensormaximumtensorrAr%rrHrIr"rJ) rr%rrrKrLr<r>rMr:rNrr(r(r)_compute_dynamic_ntk_parameterss&+    &,rVcs||dur |j|n|j}|d}|dd}t|d|j|j}t||}|d} |d} |d} |d } |d } | durJ|j| } dd d }| durh| rd| rdt|| | || | } n|| } |dpnd}|dpud }ddfdd}dd}|t d|dj |t jd|}d|}d| |}|jdd}|||||| |\}}d ||||dj |t jd}|d |||}|| fS)a  Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as available. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunciton with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. Nr;r<r=r>r:rNmscalemscale_all_dimr rcSs"|dkrdSd|t|dS)Nrr=g?)mathlog)scalerWr(r(r) get_mscaleNsz,_compute_yarn_parameters..get_mscale beta_fast beta_slowcSs*|t||dtjdt|S)zPInverse dimension formula to find the dimension based on the number of rotationsr?)rYrZpi) num_rotationsrMrLrQr(r(r)find_correction_dim`s*z5_compute_yarn_parameters..find_correction_dimcsL||||}||||}|rt|}t|}t|dt||dfS)z.Find dimension range bounds based on rotationsrr)rYfloorceilrmin)low_rothigh_rotrMrLrQtruncatelowhighrbr(r)find_correction_rangeds   z7_compute_yarn_parameters..find_correction_rangecSs>||kr|d7}tj|tjd|||}t|dd}|S)NgMbP?r@rr)rrHfloat32clamp)rerrM linear_func ramp_funcr(r(r)linear_ramp_factorms z4_compute_yarn_parameters..linear_ramp_factorrr?rBrhT)r) rCrrDrrErFrGrQrJrrHr")rr%rrrKrLr<r>rMr:rNrWrXr r\r]r_rlrq pos_freqsinv_freq_extrapolationinv_freq_interpolationrhrirjinv_freq_extrapolation_factorrr(rkr)_compute_yarn_parameterssD:         "    rvcCs.||dur |j|n|j}|d}|dd}t|d|j|j}t||}|d} |d} |d} |d } |d } | durI|j| } | durc| dkrTd} nt d t | t | } |rs|| krst j | t j |d }n t j | t j |d }t jd |dt j|d |}d|||}|| fS)a Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. Nr;r<r=r> long_factor short_factorr:rNr rrPrr?)rCrrDrrErFrGrQrYsqrtrZrrUrmrHrIrJ)rr%rrrKrLr<r>rMrwrxr:rNr ext_factorsinv_freq_shaperr(r(r)_compute_longrope_parameterss.2      r|cCs2||dur |j|n|j}|d}|dd}t|ddp%|j|j}t||}d} d|tjd|dtj dj |tj d |} |d } |d } |d } |d }|| }|| }dt j | }t||k| | | }||| | | }d||| ||}||k||k}t|||}|| fS)au Computes the inverse frequencies for llama 3.1. Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `factor` (`float`, *optional*): The scaling factor applied to the inverse frequencies when 1) the wavelength is greater than `low_freq_wavelen` prior to smoothing, and 2) to all inverse frequencies during smoothing. * `high_freq_factor` (`float`): The scale factor used to compute `high_freq_wavelen` and the value for the denominator of the smoothing factor prior to the `low_freq_factor` shift. * `low_freq_factor` (`float`): The scale factor used to compute `low_freq_wavelen` and the shift applied to the numerator and denominator of the smoothing factor. frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. If not provided, the function falls back to `max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. Nr;r<r=r>rr?r@rBr:low_freq_factorhigh_freq_factorr r)rCrrDrrErFrGrrHrIr"rJrYr`where)rr%rrrKrLr<r>rMrNrr:r}r~old_context_lenlow_freq_wavelenhigh_freq_wavelenwaveleninv_freq_llama smooth_factorsmoothed_inv_freqis_medium_freqr(r(r)_compute_llama3_parameterss*-  ,r)linearr0yarnr1llama3c@seZdZUdZeed<edBed<edBed<edBed<edBed<edBed<edBed <edBed <eedBed <eedBed <edBed <edBed<dS)RopeParametersaY Args: rope_theta (`float`): The base period of the RoPE embeddings. rope_type (`str`, *optional*, defaults to "default"): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. partial_rotary_factor (`float`, *optional*): The percentage of the query and key head embedding on which RoPE will be applied. factor (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. original_max_position_embeddings (`int`, *optional*): Used with 'yarn', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. attention_factor (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. beta_fast (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. beta_slow (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. short_factor (`list[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 long_factor (`list[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 low_freq_factor (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE high_freq_factor (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE r;Nrr<r:r rNr]r_rxrwr}r~) __name__ __module__ __qualname____doc__rJ__annotations__strrGlistr(r(r(r)r:s (        rF)totalc @seZdZdZdZd!dedBfddZddZd!d d d edBfd d Zd!de d edBfddZ d!de d edBfddZ d!de d edBfddZ d!de d edBfddZ d!de d edBfddZd!de d edBfddZe  d"dededededBd edBf dd ZdS)#RotaryEmbeddingConfigMixinz[ A Mixin containing the functionality to standardize and validate RoPE parameters. g@Nignore_keys_at_rope_validationcKs|dd}|p |j|_|jdur|jni|_|dt|d|j}|jd||dt|dd}|durL|jd||durEtn|}|dhB}||j|d|S)N rope_scalingr;r< ignore_keys) poprr default_theta setdefaultrDsetrC validate_rope)r#rr3rr;r<r(r(r)convert_rope_params_to_dictxs    z6RotaryEmbeddingConfigMixin.convert_rope_params_to_dictcCsNt|dd}t|dd}t|ddpi}t|dd}|s%|s%tddS|dus6|iks6t||si|d|dd |d||durN||d<|dd vrht|d r`|j |j d <nB|j d |j n9t|D]4}||d||dd ||d||dur|||d<||dd vr|j |d |j qm||_ dS) z Helper to standardize the config's rope params field by ensuring the params are defined for each later type. For old model the fn will duplicate a single rope param in each layer type (backward compatibility) r;Nr<r layer_typeszG`standardize_rope_params` was called but no RoPE parameters were found.rtypedefault)rrr1r ) rloggerwarningrkeysissubsetrrDrr rrQ)r#r;r<rrrr(r(r)rCs8    "       z2RotaryEmbeddingConfigMixin.standardize_rope_paramsr#r rcCs|j}|dur dSt|dddurt||jrnd|i}|D].}|d|dd}t|d|dd}||d<|durI|||d q$t d |d q$dS) zY Validate the RoPE config arguments, given a `"PreTrainedConfig"` object Nrfull_attentionrrr _validate__rope_parametersrzMMissing validation function in 'RotaryEmbeddingConfigMixin' for 'rope_type'='') rrrrrrvaluesrDrr)r#rrKrr validation_fnr(r(r)rs$  z(RotaryEmbeddingConfigMixin.validate_ropercCs2ddh}t|}|d}|j||||ddS)Nrr;r)rr_check_received_keys)r#rr required_keys received_keysrr(r(r)!_validate_default_rope_parameterss zr:rr;rrr:r=;`rope_parameters`'s factor field must be a float >= 1, got rrrrRrJrrr#rrrrrr:r(r(r) _validate_linear_rope_parameters z;RotaryEmbeddingConfigMixin._validate_linear_rope_parameterscCshddh}t|}|d}|j||||d|d}|dus(t|tr(|dkr2td|dSdS)Nrr:rr=rrrr(r(r)!_validate_dynamic_rope_parametersrzr:rr;r >rWrhr]r_rXrNrrr:r=rrNrO`rope_parameters`'s attention_factor field must be a float greater than 0, got r]z9`rope_parameters`'s beta_fast field must be a float, got r_z9`rope_parameters`'s beta_slow field must be a float, got r^rzR`rope_parameters`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r zKThe explicitly set RoPE scaling factor (config.rope_parameters['factor'] = z) does not match the ratio implicitly set by other parameters (implicit factor = post-yarn context length / pre-yarn context length = config.max_position_embeddings / config.rope_parameters['original_max_position_embeddings'] = z). Using the explicit factor (z) in YaRN. This may cause unexpected behaviour in model usage, please correct the 'original_max_position_embeddings' fields in the model config.) rrrrRrJrrrDrrQ warning_once) r#rrr optional_keysrrr:rNr]r_r implicit_factorr(r(r)_validate_yarn_rope_parameterssH      z9RotaryEmbeddingConfigMixin._validate_yarn_rope_parameterscCshd}ddh}t|}|d}|j|||||d|dd}t|d|j|j}t||} |d } t| t sMt d d | DrMt d | t | | d krdt d| d dt | |d} t| t st dd | Drt d| t | | d krt d| d dt | |d} |d} | dur| durt dn| dur| durt dnt| tr| dkrt d| |d}|durt|tr|dkrt d|dSdSdS)N>rr;rwrxr rNr:rrr<r=r>rxcs|] }t|ttfVqdSr7rRrGrJ.0r2r(r(r) 5zPRotaryEmbeddingConfigMixin._validate_longrope_rope_parameters..zF`rope_parameters`'s short_factor field must be a list of numbers, got r?z8`rope_parameters`'s short_factor field must have length z, got rwcsrr7rrr(r(r)r=rzE`rope_parameters`'s long_factor field must be a list of numbers, got z7`rope_parameters`'s long_factor field must have length r avThis model config has set a `rope_parameters['original_max_position_embeddings']` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_parameters`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.z4Missing required keys in `rope_parameters`: 'factor'rgr)rrrrDrrErFrGrRrallrrlenrrJ)r#rrrrrrr<r>rMrxrwr:r rNr(r(r)"_validate_longrope_rope_parameters)sL        z=RotaryEmbeddingConfigMixin._validate_longrope_rope_parametersc Cs&hd}|d}t|}|j||||d|d}|dus(t|tr(|dkr0td||d}|d}|dusAt|tsItd ||dusRt|tsZtd |||kritd |d ||d } | dusvt| ts~td| | |jkrtd| d|jdSdS)N>r:rr;r}r~r rrr:r=rr}r~z?`rope_parameters`'s low_freq_factor field must be a float, got z@`rope_parameters`'s high_freq_factor field must be a float, got zf`rope_parameters`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zS`rope_parameters`'s original_max_position_embeddings field must be an integer, got zj`rope_parameters`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rrrrRrJrrrGrQ) r#rrrrrr:r}r~r r(r(r) _validate_llama3_rope_parameters[sJ  z;RotaryEmbeddingConfigMixin._validate_llama3_rope_parametersrrrrcCsd|vr|dh8}|d|pt}d|vr|d|dur$||8}||}|r4td|d||||}|rItd|d|dSdS)z\Compare the received keys in `config.rope_parameters` against the expected and optional keysrrr<Nz