o iA@sddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z ddlmZddlmZddlmZmZmZmZddlmZmZdd lmZerjdd lmZdd lmZm Z m!Z!m"Z"ee#$j%Z&e&j%Z'Gd d d e Z(dZ)dOddZ*edZ+edZ,dPddZ-dQdd Z.ej/dRd$d%Z0ej1dSd(d)Z2dTd,d-Z3Gd.d/d/Z4 dUdVd4d5Z5 6dWdXd;d<Z6dYd@dAZ7dZdCdDZ8d[dIdJZ9GdKdLdLZ:GdMdNdNee+Z;dS)\) annotationsN) is_dataclass)autoEnum)Pathpformat)AnyGeneric TYPE_CHECKINGTypeVar) assert_neverSelf) CodeTemplate) Namespace)CallableIterableIteratorSequencec@s0eZdZeZeZeZeZeZeZ dS)TargetN) __name__ __module__ __qualname__r DEFINITION DECLARATION REGISTRATIONANONYMOUS_DEFINITIONNAMESPACED_DEFINITIONNAMESPACED_DECLARATIONrrP/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/torchgen/utils.pyr%s rz(^|\W){}($|\W)schemastrreturntuple[str, list[str]]cCs>td|}|durtd||\}}}||dfS)Nz(\w+)(\.\w+)?\((.*)\)zUnsupported function schema: , )rematch RuntimeErrorgroupssplit)r!mname_paramsrrr split_name_params8s r/TSfuncCallable[[T], S | None]xs Iterable[T] Iterator[S]ccs&|D] }||}|dur|VqdSNr)r2r4xrrrr mapMaybeHsr:Callable[[T], Sequence[S]]ccs|D] }||EdHqdSr7r)r2r4r8rrr concatMapPsr<msg_fnCallable[[], str]Iterator[None]c cspzdVWdSty7}z$|}t|d}|jr%|jdd|n|}|f|jdd|_d}~ww)Nz r ) Exceptiontextwrapindentargs)r=emsgrrr contextXs  rH template_fnrcCs t|Sr7)r from_file)rIrrr _read_templatees rKsintcCs&tj|ddd}tj|ddS)Nlatin1F)usedforsecuritylittle) byteorder)hashlibsha1encodedigestrM from_bytes)rLrSrrr string_stable_hashksrWc@seZdZd1dd Zed2d d Zd3ddZd4ddZd5ddZd6ddZ ddd7d(d)Z ddd8d*d+Z d9d-d.Z d:d/d0Z dS); FileManager install_dir str | Path template_dirdry_runboolr#NonecCs&t||_t||_t|_||_dSr7)rrYr[setfilesr\)selfrYr[r\rrr __init__ts   zFileManager.__init__frozenset[str]cCstdd|jDS)NcSsh|]}|qSras_posix.0filerrr sz(FileManager.filenames..) frozensetr`rarrr filenamesszFileManager.filenamesfilenamecontentsr"cCs^t|}d}z|jdd}Wn tyYnw||kr-|jjddd|j|dddSdS)Nzutf-8)encodingT)parentsexist_ok)r read_textOSErrorparentmkdir write_text)rarmrnrh old_contentsrrr _write_if_changeds zFileManager._write_if_changedrI env_callable"Callable[[], str | dict[str, Any]]c Cs t|r Jd||j|}|}t|tryd|vrftd}zttjdjp+| }Wnt t fy?| }Ynwz | t }Wn tyU|j}Ynwi|ddd|d|i}t|}||} tjdd | tjd St|tr|St|dS) Nztemplate_fn must be relative: generated_commentzgen.py__main__@z generated by z from z (""")\n+ # match triple quotes ( (\s*@.+\n)* # match decorators if any \s*(class|def) # match class/function definition ) z\g<1>\n\n\g<2>)flags)r is_absoluter[ isinstancedict TORCHGEN_ROOTsysmodules__file__absoluteKeyErrorAttributeError relative_to REPO_ROOTre ValueErrorr,rK substituter&subVERBOSEr"r ) rarIry template_pathenvgenerator_default generatorgenerator_pathtemplatesubstitute_outrrr substitute_with_templatesN        z$FileManager.substitute_with_templatecCsvt|}|rJd||j|}||jvs Jd||j||js9|j||d}|j||ddSdS)Nfilename must be relative: zduplicate file write )rIry)rmrn)rrrYr`addr\rrx)rarmrIryrhrrrr write_with_templates  zFileManager.write_with_templatecCs||||dSr7)r)rarmryrrr writeszFileManager.writeN)base_envitemsr5key_fnCallable[[T], str]#Callable[[T], dict[str, list[str]]] num_shardsrMrdict[str, Any] | None sharded_keysset[str]c Cs|j||||||||ddS)N)rryrrr)write_sharded_with_template)rarmrrryrrrrrr write_shardeds  zFileManager.write_shardedcs\t|} | rJd|ddi} ddt|D} | g| } |dur/| D]|q'D]#} | D]| vrOt| tsFJd| | <q5g| <q5q1dfd d }|jrad}|D]}||} t| |}||}|| |||| |qc| D]d}| | | j ||fddq|j |j| | j ddS)Nrshard_id EverythingcSsg|] }dd|iqS)rr-r)rgirrr sz;FileManager.write_sharded_with_template..z'sharded keys in base_env must be a listintodict[str, list[str]]from_r#r^cs<|D]\}}|vsJd||||7<qdS)Nzundeclared sharded key )r)rrkv)rrr merge_envsz:FileManager.write_sharded_with_template..merge_envrcsSr7rr)shardrr -sz9FileManager.write_sharded_with_template..)rrrrr#r^)rrrangeupdaterlistcopyr\rWr with_stemstemr`discardrY)rarmrIrrryrrrrh everythingshards all_shardskeyritemsidrrr)rrr rsH        $z'FileManager.write_sharded_with_template variable_namecCs8dd|gddt|jDdR}|||dS)zSWrite a file containing the list of all outputs which are generated by this script.r@zset(css |] }d|dVqdS)z ""Nrdrfrrr :sz,FileManager.write_outputs..)N)joinsortedr`rx)rarrmcontentrrr write_outputs3s zFileManager.write_outputscCstj|jtjtS)z This needs to be deterministic. The template dir is an absolute path that varies across builds. So, just use the path relative to this file, which will point to the codegen source but will be stable. )ospathrelpathr[dirnamerrkrrr template_dir_for_comments@sz%FileManager.template_dir_for_comments)rYrZr[rZr\r]r#r^)r#rc)rmrZrnr"r#r^)rIrZryrzr#r")rmrZrIrZryrzr#r^)rmrZryrzr#r^)rmrZrr5rrryrrrMrrrrr#r^)rmrZrIrZrr5rrryrrrMrrrrr#r^)rr"rmrZr#r^r#r")rrrrbpropertyrlrxrrrrrrrrrrr rXss    6  > rXoptionsrrYstr | Path | NonecCs.tj|jd}|r |n|j}t|||jdS)N templates)rYr[r\)rrr source_pathrYrXr\)rrYr[rrr make_file_managerJsrPobjr rDwidthcCs t|||Sr7r)rrDrrrr dataclass_reprXs rattrdict[Any, Any] curr_indentc Csv||d7}g}|D]%\}}t|}t|r#t||||t|nt|}||d|q t||||ddS)Nz: {})rreprrrlenappend_format) rrDrr dict_reprrrk_reprv_strrrr _format_dict`s r&list[Any] | set[Any] | tuple[Any, ...]csJd7fdd|D}t|trdnd\}}t|||S)NrAcs*g|]}t|rt|nt|qSr)rrr)rglrrDrrr r{sz _format_list..)[])(r)rrr)rrDrr list_reprstartendrrr _format_listts r fields_str list[str]rrc CsTd\}}tt||krd}d|}d|}d|||} ||| |S)N)rr@ r%)rrr) rrDrrrr delimitercurr_indent_str indent_strbodyrrr rsrc@sheZdZdZ  ddd d Ze ddddZedddZedddZ edddZ dd ddZ dS)!NamespaceHelperaA helper for constructing the namespace open and close strings for a nested set of namespaces. e.g. for namespace_str torch::lazy, prologue: namespace torch { namespace lazy { epilogue: } // namespace lazy } // namespace torch r namespace_strr" entity_name max_levelrMr#r^cCsr|d}t||ksJd|d|d||_ddd|D|_dddt|D|_||_||_dS) N::z"Codegen doesn't support more than z# level(s) of custom namespace. Got .r@cSsg|]}d|dqS)z namespace z {rrgnrrr rsz,NamespaceHelper.__init__..cSsg|]}d|qS)z} // namespace rrrrr rs) r*rcpp_namespace_r prologue_reversed epilogue_ namespaces_ entity_name_)rarrrcpp_namespacesrrr rbs  zNamespaceHelper.__init__namespaced_entitycCs2|d}|d}d|dd}t|||dS)zq Generate helper from nested namespaces as long as class/function name. E.g.: "torch::lazy::add" rN)rrr)r*rr)rrnamesrrrrr from_namespaced_entitys z&NamespaceHelper.from_namespaced_entitycC|jSr7)rrkrrr prologuezNamespaceHelper.prologuecCr r7)rrkrrr epiloguer zNamespaceHelper.epiloguecCr r7)rrkrrr rr zNamespaceHelper.entity_namedefaultcCs|jr|jS|S)z Return the namespace string from joining all the namespaces by "::" (hence no leading "::"). Return default if namespace string is empty. )r)rarrrr get_cpp_namespacesz!NamespaceHelper.get_cpp_namespaceN)rr)rr"rr"rrMr#r^)r)rr"rrMr#rr)r)rr"r#r") rrr__doc__rb staticmethodr rr rrrrrrr rs    rc@seZdZUded<d&d'dd Zd(d dZd)ddZd*ddZd+ddZd,ddZ e d-ddZ d.ddZ d/d!d"Z d0d$d%ZdS)1 OrderedSetz dict[T, None]storageNiterableIterable[T] | Noner#r^cCs"|dur i|_dSt||_dSr7)rrfromkeys)rarrrr rbs zOrderedSet.__init__rr0r]cCs ||jvSr7rrarrrr __contains__s zOrderedSet.__contains__ Iterator[T]cCst|jSr7)iterrkeysrkrrr __iter__zOrderedSet.__iter__r OrderedSet[T]cCs|j|jdSr7)rr)rarrrr rszOrderedSet.updatecCsd|j|<dSr7rrrrr rrzOrderedSet.addcCst}|j|_|Sr7)rrr)raretrrr rs zOrderedSet.copyrEcGs,|d}|ddD]}||q |S)NrrA)rr)rEr!rLrrr unions  zOrderedSet.unionothercCs t||Sr7)rr"rar#rrr __or__s zOrderedSet.__or__rcCs|||Sr7)rr$rrr __ior__s zOrderedSet.__ior__objectcCs(t|tr |j|jkSt|j|kSr7)rrrr_rr$rrr __eq__s  zOrderedSet.__eq__r7)rrr#r^)rr0r#r])r#r)rr r#r^)rr0r#r^)r#r )rEr r#r )r#r r#r )r#r r#r)r#r'r#r])rrr__annotations__rbrrrrrrr"r%r&r(rrrr rs          r)r!r"r#r$)r2r3r4r5r#r6)r2r;r4r5r#r6)r=r>r#r?)rIr"r#r)rLr"r#rMr7)rrrYrr#rX)rr)rr rDrMrrMr#r") rrrDrMrrMrrMr#r") rrrDrMrrMrrMr#r")rrrDrMrrMrrMrr"rr"r#r")< __future__r contextlib functoolsrRrr&rrC dataclassesrenumrrpathlibrpprintrtypingr r r r typing_extensionsr rtorchgen.code_templaterargparsercollections.abcrrrrrrrtrrr IDENT_REGEXr/r0r1r:r<contextmanagerrHcacherKrWrXrrrrrrrrrrr sX             Z    E