o i5 @sddlZddlZddlZddlmZddlmZmZddlm Z ddl Z ddl m m mmZddlmZddlmZmZddlmZddlmZdd lmZmZmZdd lmZdd l m!Z!dd l"m#Z#dd l$m%Z%ddl&m'Z'gdZ(e%e)Z*eGdddZ+GdddZ,dee-BdBde.e de-fddZ/de!de0e-dBe1dBffddZ2de+dee-BdBde.e de3e1e ffddZ4dS)N)Callable) dataclassfield)Any)get_default_numa_options)eventsmetrics) WorkerSpec)LocalElasticAgent)DefaultLogsSpecs LogsSpecsSignalException)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) NumaOptions) LaunchConfigelastic_launch launch_agentc@sReZdZUdZeed<eed<eed<dZedBed<dZe ed<d Z e ed <dZ e ed <d Z e ed <e edZee efed<dZeed<dZeed<dZeed<dZe ed<dZe dBed<e edZee e fed<dZe dBed<dZe ed<dZedBed<dZe ed<dZee dBed <dZee dBed!<d"Z e!ed#<d$d%Z"dS)&ra Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. local_ranks_filter: ranks for which to show logs in console. If not set, show from all. event_log_handler: name of the event logging handler as registered in `elastic/events/handlers.py `_. duplicate_stdout_filters: If non-empty, duplicates stdout to a file containing only lines that match _any_ of the filter strings. duplicate_stderr_filters: If non-empty, duplicates stderr to a file containing only lines that match _any_ of the filter strings. virtual_local_rank: Enable virtual local rank mode for workers (defaults to False). When enabled, LOCAL_RANK is set to 0 for all workers and CUDA_VISIBLE_DEVICES is adjusted so each worker accesses its assigned GPU at device index 0. .. note:: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_nodeN logs_specsrun_id default_rolerole rdzv_endpointetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsg?monitor_intervalspawn start_methodlog_line_prefix_template metrics_cfg local_addrnullevent_log_handler numa_optionszSIGTERM,SIGINT,SIGHUP,SIGQUITsignals_to_handleduplicate_stdout_filtersduplicate_stderr_filtersFvirtual_local_rankcCsd}|jdkr|j|jd<n d|jvr||jd<|jdur!t|_|jdur@tjrBtj|j krDt |_t d|jdSdSdSdS)Nir#timeoutzUsing default numa options = %r) r$r"rr r/torchcuda is_available device_countrrloggerinfo)selfdefault_timeoutr=`/sda-disk/www/egybert/egybert_env/lib/python3.10/site-packages/torch/distributed/launcher/api.py __post_init__qs     zLaunchConfig.__post_init__)#__name__ __module__ __qualname____doc__int__annotations__rr rstrrrr rdictr"rr$r&r'floatr)r*r+r,r.r/rr0r1listr2r3boolr?r=r=r=r>r's2 1            rc@s2eZdZdZdedeeBdBfddZddZdS) ra Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") config entrypointNcCs||_||_dSN)_config _entrypoint)r;rKrLr=r=r>__init__s zelastic_launch.__init__cGst|j|jt|SrM)rrNrOrI)r;argsr=r=r>__call__szelastic_launch.__call__) r@rArBrCrrrFrPrRr=r=r=r>rs  rrLrQreturncCs@t|tr|jSt|tr|tjkrtdd|DdS|SdS)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. css |] }|ddkr|VqdS)r-Nr=).0argr=r=r> sz'_get_entrypoint_name..r) isinstancerr@rFsys executablenext)rLrQr=r=r>_get_entrypoint_names   r\rdzv_parameterscCsX|jdkrdS|j}|}|stdt|dd\}}|dkr(td|d||fS)Nstatic)NNzKEndpoint is missing in endpoint. Try to add --master-addr and --master-portr#) default_portzport is missing in endpoint: z. Try to specify --master-port)backendendpointstrip ValueErrorr)r]ra master_addr master_portr=r=r>_get_addr_and_ports  rfrKc Cs.|jsttj}td|||_t||}tdid|d|j d|j d|j d|jd|j d |j d |jd |jd |jd |jjd|jd|jd|jd|jd|jd|jtd|j |j |j|j |j |jd|j}t|\}}|jtjd<t|j|j |t |t!"||j|j|||j|j|j|j|j|j#d}t$||j|j%|j&d} d} zZz.t'(t')|j| *} t+,| -|j| .rt/|| j0d| j1WW| r|j23SSt/yt4yd} t+,| 5|jt6y t+,| 5|jw| r|j23ww)Nz3config has no run_id, generated a random run_id: %saRStarting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s event_log_handler : %(event_log_handler)s numa_options : %(numa_options)s signals_to_handle : %(signals_to_handle)s duplicate_stdout_filters : %(duplicate_stdout_filters)s duplicate_stderr_filters : %(duplicate_stderr_filters)s rLrrrrr rr"r&r'log_dirr+r.r/r0r1r2)r`rarrrr,TORCHELASTIC_SIGNALS_TO_HANDLE)rlocal_world_sizerLrQ rdzv_handlerr&r'rdrer,r.r/r1r2r3)specrr)r*T)namefailuresFr=)7rrFuuiduuid4rDr9warningr\r:rrrr rr"r&r'r root_log_dirr+r.r/r0r1r2rr,rfosenvironr rtuple rdzv_registryget_rendezvous_handlerr3r r)r*rinitialize_metrics MetricsConfigrunrrecordget_event_succeeded is_failedrrm return_valuesrjshutdownr get_event_failed Exception) rKrLrQrentrypoint_namer]rdrerkagent shutdown_rdzvresultr=r=r>rs       (     r)5rrrYrncollections.abcr dataclassesrrtypingrr5-torch.distributed.elastic.rendezvous.registry distributedelastic rendezvousregistryrutorch._utils_internalrtorch.distributed.elasticrr*torch.distributed.elastic.agent.server.apir :torch.distributed.elastic.agent.server.local_elastic_agentr )torch.distributed.elastic.multiprocessingr r r 0torch.distributed.elastic.multiprocessing.errorsr$torch.distributed.elastic.rendezvousr*torch.distributed.elastic.rendezvous.utilsr'torch.distributed.elastic.utils.loggingrtorch.numa.bindingr__all__r@r9rrrFrIr\rtrDrfrGrr=r=r=r>sL          ^"'