o iP @sxUddlZddlmZmZmZddlmZddlmZm Z ddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZddlZd dgZe dZedZejjZddZiZeeefe d<ddZ!dVde e eefge eefffddZ"e"ej#ddde$fddZ%e"ej&dWde$fddZ'e"ej(dWde$fddZ)e"ej*dWde$fd d!Z+e"ej,     dXde$fd"d#Z- dVd$e.e$d%e.e$d&e.e$d'e/de$f d(d)Z0e"ej1ej2ej3ej4ej5gddde$fd*d+Z6e"ej7de$fd,d-Z8d.d/Z9e"ej:ej;ejdd4de e?e?e$d5fe?e$d5fe?e$d5fe?e$d5fdBffd6d7Z@dd4de e?e?e$d5fe?e$d5fe?e$d5fe?e$d5fdBffd8d9ZAe"ejBd:d;ddde$fdd?ZEd@dAZFe"ejGejHejIgddde$fdBdCZJe"ejKd:d;de$fdDdEZLe"ejMd:d;de$fdFdGZNiej#e%ej&e'ej(e)ej*e+ej,e-ej1e6ej2e6ej3e6ej5e6ej4e6ej7e8ej:e=ej;e=ej.nfrr%r&rr$r shape_wrappersr(Freturncs,dtttfdtttfffdd }|S)N flop_formular)cs.stdfdd }tjj|S)Nr)csHt|tjjstd|dt||tvrtd|t|<dS)Nzlregister_flop_formula(targets): expected each target to be OpOverloadPacket (i.e. torch.ops.mylib.foo), got z which is of type zduplicate registrations for )rr_opsOpOverloadPacket ValueErrortyper RuntimeError)targetr*rrregister)s z=register_flop_formula..register_fun..register)r)N)r(rutils_pytree tree_map_)r*r2get_rawtargetsr1r register_fun%s  z+register_flop_formula..register_fun)r rr)r8r7r9rr6rr$s()r!c Os<|\}}|\}}||krtd|d|||d|S)zCount flops for matmul.z3matmul: inner dimensions must match (k == k2), got  and AssertionError) a_shapeb_shaper!r"r#mkk2nrrrmm_flop:s rDcK t||S)zCount flops for addmm.rD self_shaper>r?r!r#rrr addmm_flopFs rIc Ksd|\}}}|\}}} ||krtd|d|||kr&td|d|||| d|} | S)z"Count flops for the bmm operation.z0bmm: batch dimensions must match (b == b2), got r:z0bmm: inner dimensions must match (k == k2), got r;r<) r>r?r!r#br@rAb2rBrCfloprrrbmm_flopKs  rMcKrE)z&Count flops for the baddbmm operation.)rMrGrrr baddbmm_flopZs rNc Ks t||S)zCount flops for _scaled_mm.rF) r>r? scale_a_shape scale_b_shape bias_shapescale_result_shape out_dtypeuse_fast_accumr!r#rrr_scaled_mm_flopas rUx_shapew_shaper! transposedc CsL|d}|r|n|dd}|^}}} t|t||||d} | S)aCount flops for convolution. Note only multiplication is counted. Computation for bias are ignored. Flops for a transposed convolution are calculated as flops = (x_shape[2:] * prod(w_shape) * batch_size). Args: x_shape (list(int)): The input shape before convolution. w_shape (list(int)): The filter shape. out_shape (list(int)): The output shape after convolution. transposed (bool): is the convolution transposed Returns: int: the number of flops rr;Nr) rVrWr!rX batch_size conv_shapec_outc_in filter_sizerLrrrconv_flop_countrs   r^c Ost||||dS)zCount flops for convolution.rX)r^) rVrW_bias_stride_padding _dilationrXr!r"r#rrr conv_flopsrdc Csdd} d} | drt| d}| t|||| 7} | drIt| d}|r9| t| || || |dd7} | S| t| || || |dd7} | S)NcSs |d|dgt|ddS)Nrrr;)list)rrrrts zconv_backward_flop..trrFr_)rr^)grad_out_shaperVrWr`rarbrcrX_output_padding_groups output_maskr!rf flop_countgrad_input_shapegrad_weight_shaperrrconv_backward_flopsF    rncCs|\}}}}|\}}} } |\} } } }||kr| kr   z0_flash_attention_forward_flop..rsum) rrrrrrrr!r"r#sizesrrr_flash_attention_forward_flop rc Os(t|||||||d} tdd| DS)r)rrrrrrrcsrrrrrrrrrz4_efficient_attention_forward_flop..rr) rrrbiasrrrrr"r#rrrr!_efficient_attention_forward_floprrcCsfd}|\}}}}|\} } } } |\} }}}|\}}}}|| kr)| kr)|krFntd|| kr=|kr=|krFntd|| ksJtd||krV| |krV||ksZtdd}|t||||f|||| f7}|t||||f|||| f7}|t||| |f||||f7}|t|||| f||| |f7}|t||||f|||| f7}|S)NrzFsdpa_backward_flop_count: batch/heads/dimension mismatch among tensorszJsdpa_backward_flop_count: grad_out/value/key/query shapes are incompatiblero)rgrprqrrr~rJrsrtrurvrwrxryrzr{r|r}_b4_h4_s4_d4rrrsdpa_backward_flop_counts(      """""rcOst||||S)z(Count flops for self-attention backward.r)rgrprqrrr!r"r#rrrsdpa_backward_flopsrc O*t|||||||| d} tdd| DS)N)rrrrrrrrcs&|]\}}}}t||||VqdSrrrrprqrrrgrrrr   z1_flash_attention_backward_flop..r) rrrrout logsumexprrrrr"r#shapesrrr_flash_attention_backward_flop rc Or)N)rrrrrrrrcsrrrrrrrr>rz5_efficient_attention_backward_flop..r) rrrrrrrrrrr"r#rrrr"_efficient_attention_backward_flop#rrcCst|ts|fS|Sr)rtuple)xrrrnormalize_tuple\s r)KMBTcCs0tdtttdtt|dd}t|S)Nrrr;r)maxminrsuffixesstr)numberindexrrrget_suffix_stres(rcCs&t|}|d|d}|t|S)Niz.3f)rr)rsuffixrrrrrconvert_num_with_suffixls  rcCs|dkrdS||dS)Nr0%z.2%r)numdenomrrrconvert_to_percent_strss rcstfdd}|S)Ncst|\}}|}t||Sr)rr)r" flat_argsspecrr$rrr&ys  z)_pytreeify_preserve_structure..nfrr'rr$r_pytreeify_preserve_structurexsrc seZdZdZ    ddejjeejjBdBdede de e e fdBd df fd d Z d efd d Z d e ee e efffddZdddZddZddZddZZS)ra ``FlopCounterMode`` is a context manager that counts the number of flops within its context. It does this using a ``TorchDispatchMode``. It also supports hierarchical output by passing a module (or list of modules) to FlopCounterMode on construction. If you do not need hierarchical output, you do not need to use it with a module. Example usage .. code-block:: python mod = ... with FlopCounterMode(mod) as flop_counter: mod.sum().backward() Nr;Tmodsdepthdisplaycustom_mappingr)cstttdd|_||_||_d|_|duri}|dur&tjdddit dd| D|_ t |_ dS)NcSsttSr)r intrrrrsz*FlopCounterMode.__init__..zs*z,FlopCounterMode.__init__..) super__init__r flop_countsrrmodewarningswarnritemsr mod_tracker)selfrrrr __class__rrrs  zFlopCounterMode.__init__cCst|jdS)NGlobal)rrvaluesrrrrget_total_flopsszFlopCounterMode.get_total_flopscCsdd|jDS)aReturn the flop counts as a dictionary of dictionaries. The outer dictionary is keyed by module name, and the inner dictionary is keyed by operation name. Returns: Dict[str, Dict[Any, int]]: The flop counts as a dictionary. cSsi|] \}}|t|qSr)dictrrrrrsz3FlopCounterMode.get_flop_counts..)rrrrrrget_flop_countss zFlopCounterMode.get_flop_countsc s|durj}|dur d}ddl}d|_gd}g}tdfdd}tjD]}|dkr;q4|d d }||krGq4|||d }| |q4djvroso|D] } d | d| d<q]|dd|}t |dkrzgd g}|j||d dS)Ni?BrT)ModuleFLOPz% TotalFcstj|}|kOd|}g}|||t|t|gj|D]\}}||dt|t|t|gq,|S)N z - )rrrappendrrrr)mod_namerr~paddingrrAr global_flops global_suffixis_global_subsumedrrr process_mods  z.FlopCounterMode.get_table..process_modr.rr )r0r)leftrightr)headerscolalign) rtabulatePRESERVE_WHITESPACErrsortedrkeyscountextendr) rrrheaderrrmod mod_depth cur_valuesrrrr get_tables6   zFlopCounterMode.get_tablecCs,|j|jt||_|j|Sr)rclearr __enter___FlopCounterModerrrrrr%s    zFlopCounterMode.__enter__cGsH|jdur td|jj|}d|_|j|jr"t||j|S)Nzs&         *   & g6 96 7