&VjilZUddlmZddlZddlZddlmZmZmZmZmZddl m Z m Z m Z m Z ddlZddlmZerddlmZddlmZgd Ze d Ze d Zeejd sHed ejjd <edejjd<edejjd<ddlmZmZmZd/dZd0dZGddejjZ GddZ!edede"ffZ#de$d<e d1d2d(Z%e d1d3d+Z% d1d4d.Z%dS)5) annotationsN)CallableOptionaloverload TYPE_CHECKINGUnion) ParamSpecSelf TypeAliasTypeVar)Tensor) _POOL_HANDLE) _dummy_type)is_current_stream_capturinggraph_pool_handle CUDAGraphgraphmake_graphed_callables_R_P_CudaStreamBase _CUDAGraph_graph_pool_handle_cuda_isCurrentStreamCapturing)rrrreturnboolctS)zReturn True if CUDA graph capture is underway on the current CUDA stream, False otherwise. If a CUDA context does not exist on the current device, returns False without initializing the context. )rK/root/voice-cloning/.venv/lib/python3.11/site-packages/torch/cuda/graphs.pyrr/s * + ++r rcXtjtS)zReturn an opaque token representing the id of a graph memory pool. See :ref:`Graph memory management`. .. warning:: This API is in beta and may change in future releases. )torchcudarrrr r!rr8s! : " "#5#7#7 8 88r ceZdZdZddfd Z ddfd Zd fd Zd fd Zd fd Zd fd Z d!fd Z d fd Z d"fd Z d#fd Z d#fd ZxZS)$ra-Wrapper around a CUDA graph. Arguments: keep_graph (bool, optional): If ``keep_graph=False``, the cudaGraphExec_t will be instantiated on GPU at the end of ``capture_end`` and the underlying cudaGraph_t will be destroyed. Users who want to query or otherwise modify the underlying cudaGraph_t before instantiation can set ``keep_graph=True`` and access it via ``raw_cuda_graph`` after ``capture_end``. Note that the cudaGraphExec_t will not be instantiated at the end of ``capture_end`` in this case. Instead, it will be instantiated via an explicit called to ``instantiate`` or automatically on the first call to ``replay`` if ``instantiate`` was not already called. Calling ``instantiate`` manually before ``replay`` is recommended to prevent increased latency on the first call to ``replay``. It is allowed to modify the raw cudaGraph_t after first calling ``instantiate``, but the user must call ``instantiate`` again manually to make sure the instantiated graph has these changes. Pytorch has no means of tracking these changes. .. warning:: This API is in beta and may change in future releases. F keep_graphrrr cHt||SN)super__new__)clsr& __class__s r!r*zCUDAGraph.__new___swwsJ///r NglobalpoolOptional[_POOL_HANDLE]capture_error_modestrNonecNt||dS)aBegin capturing CUDA work on the current stream. Typically, you shouldn't call ``capture_begin`` yourself. Use :class:`~torch.cuda.graph` or :func:`~torch.cuda.make_graphed_callables`, which call ``capture_begin`` internally. Arguments: pool (optional): Token (returned by :func:`~torch.cuda.graph_pool_handle` or :meth:`other_Graph_instance.pool()`) that hints this graph may share memory with the indicated pool. See :ref:`Graph memory management`. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on these actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ )r.r0N)r) capture_begin)selfr.r0r,s r!r4zCUDAGraph.capture_beginbs)& 4zCUDAGraph.resets  r rcDtS)zReturn an opaque token representing the id of this graph's memory pool. This id can optionally be passed to another graph's ``capture_begin``, which hints the other graph may share the same memory pool. )r)r.r8s r!r.zCUDAGraph.pools ww||~~r cDtS)z/Enable debugging mode for CUDAGraph.debug_dump.)r)enable_debug_moder8s r!rAzCUDAGraph.enable_debug_modesww((***r debug_pathcFt|S)z Arguments: debug_path (required): Path to dump the graph to. Calls a debugging function to dump the graph if the debugging is enabled via CUDAGraph.enable_debug_mode() )r) debug_dump)r5rBr,s r!rDzCUDAGraph.debug_dumpsww!!*---r intcDtS)a}Returns the underlying cudaGraph_t. ``keep_graph`` must be True. See the following for APIs for how to manipulate this object: `Graph Managmement `_ and `cuda-python Graph Management bindings `_ )r)raw_cuda_graphr8s r!rGzCUDAGraph.raw_cuda_graphs ww%%'''r cDtS)aReturns the underlying cudaGraphExec_t. ``instantiate`` must have been called if ``keep_graph`` is True, or ``capture_end`` must have been called if ``keep_graph`` is False. If you call ``instantiate()`` after ``raw_cuda_graph_exec()``, the previously returned cudaGraphExec_t will be destroyed. It is your responsibility not to use this object after destruction. See the following for APIs for how to manipulate this object: `Graph Execution `_ and `cuda-python Graph Execution bindings `_ )r)raw_cuda_graph_execr8s r!rIzCUDAGraph.raw_cuda_graph_execs ww**,,,r )F)r&rrr )Nr-)r.r/r0r1rr2rr2rr)rBr1rr2)rrE)__name__ __module__ __qualname____doc__r*r4r7r:r<r>r.rArDrGrI __classcell__)r,s@r!rrDs40000000NVPPPPPPP*      ++++++......((((((----------r rcBeZdZUdZdZded< ddd ZddZddZdS)raContext-manager that captures CUDA work into a :class:`torch.cuda.CUDAGraph` object for later replay. See :ref:`CUDA Graphs ` for a general introduction, detailed use, and constraints. Arguments: cuda_graph (torch.cuda.CUDAGraph): Graph object used for capture. pool (optional): Opaque token (returned by a call to :func:`~torch.cuda.graph_pool_handle()` or :meth:`other_Graph_instance.pool()`) hinting this graph's capture may share memory from the specified pool. See :ref:`Graph memory management`. stream (torch.cuda.Stream, optional): If supplied, will be set as the current stream in the context. If not supplied, ``graph`` sets its own internal side stream as the current stream in the context. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ .. note:: For effective memory sharing, if you pass a ``pool`` used by a previous capture and the previous capture used an explicit ``stream`` argument, you should pass the same ``stream`` argument to this capture. .. warning:: This API is in beta and may change in future releases. .. _cudaStreamCaptureMode: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html#group__CUDART__STREAM_1g9d0535d93a214cbf126835257b16ba85 NOptional[torch.cuda.Stream]default_capture_streamr- cuda_graphrr.r/streamr0r1c0|jj(tj|j_|dn|f|_||n |jj|_|jJtj|j|_||_ ||_ dS)Nr) r,rSr#r$Streamr.capture_streamrU stream_ctxrTr0)r5rTr.rUr0s r!__init__zgraph.__init__s > 0 849J4E4E4G4GDN 1,BBTG (FFdn.S "...*++D,?@@$"4r rr2c:tjtjjjrt jtj|j |j j |j d|jidS)Nr0)r#r$ synchronizecompilerconfigforce_cudagraph_gcgccollect empty_cacherY __enter__rTr4r.r0)r5s r!rczgraph.__enter__s    > 3  JLLL     !!###%% Y  $6     r argsobjectcV|j|jj|dSr()rTr7rY__exit__)r5rds r!rgzgraph.__exit__s. ##%%%  $''''r )NNr-)rTrr.r/rUrRr0r1rJ)rdrerr2) rLrMrNrOrS__annotations__rZrcrgrr r!rrs:;?>>>> (,.2"* 555550    0((((((r rtorch.nn.Module.r _ModuleOrCallableF callables sample_argstuple[Tensor, ...]num_warmup_itersrEallow_unused_inputr.r/cdSr(rrlrmrorpr.s r!rrs r tuple[_ModuleOrCallable, ...]tuple[tuple[Tensor, ...], ...]cdSr(rrrs r!rrs %(Cr 7Union[_ModuleOrCallable, tuple[_ModuleOrCallable, ...]]9Union[tuple[Tensor, ...], tuple[tuple[Tensor, ...], ...]]c v)*tjr"tjrtdd}t |t s.d}|f}t jt tdf|f}n4t jt t tdfdf|}g)t||D] \}}t |tj j rt|j dkr0t|jdkrt|jdks Jdt!d|Ds Jdtjjj|} )t | t!d | Ds Jd  d )D} d |D*)*fd t-t|D} dt-t|D} dt-t|D} |t/n|}tjtjtj5t||| D]\}}}d\}}}t-|D]}tjj||}t d|D}t|dkrRtj|t d|Dt d|Dd|}|||fD]}~ dddn #1swxYwYtjg}g}t||| D]\}}}tj||5||}dddn #1swxYwYtjj |\}}|t |||g}g}ttC| tC|tC| D]Z\}}}t d|D} t d|D}d}t|dkrtj||5tj|t d|Dt d| Dd|}dddn #1swxYwYg}!d}"|D]A}#|#j"r#|!|!||"|"dz }",|!dBt |!}!|| ||!\|#|#d6d-}$g}%tI|D]\}&}|$| |&| |&*|&| |&||&| |&||&||&||& }'t |tj j r7d7d5}(|(||j%|'|j&|_&|%||%|'|r|%dSt |%S)8aAccept callables (functions or :class:`nn.Module`\ s) and returns graphed versions. Each graphed callable's forward pass runs its source callable's forward CUDA work as a CUDA graph inside a single autograd node. The graphed callable's forward pass also appends a backward node to the autograd graph. During backward, this node runs the callable's backward work as a CUDA graph. Therefore, each graphed callable should be a drop-in replacement for its source callable in an autograd-enabled training loop. See :ref:`Partial-network capture` for detailed use and constraints. If you pass a tuple of several callables, their captures will use the same memory pool. See :ref:`Graph memory management` for when this is appropriate. Arguments: callables (torch.nn.Module or Python function, or tuple of these): Callable or callables to graph. See :ref:`Graph memory management` for when passing a tuple of callables is appropriate. If you pass a tuple of callables, their order in the tuple must be the same order they'll run in the live workload. sample_args (tuple of Tensors, or tuple of tuples of Tensors): Samples args for each callable. If a single callable was passed, ``sample_args`` must be a single tuple of argument Tensors. If a tuple of callables was passed, ``sample_args`` must be tuple of tuples of argument Tensors. num_warmup_iters (int): The number of warmup iterations. Currently, ``DataDistributedParallel`` needs 11 iterations for warm up. Default: ``3``. allow_unused_input (bool): If False, specifying inputs that were not used when computing outputs (and therefore their grad is always zero) is an error. Defaults to False. pool (optional): Token (returned by :func:`~torch.cuda.graph_pool_handle` or :meth:`other_Graph_instance.pool()`) that hints this graph may share memory with the indicated pool. See :ref:`Graph memory management`. .. note:: The ``requires_grad`` state of each Tensor in ``sample_args`` must match the state that's expected for the corresponding real input in the training loop. .. warning:: This API is in beta and may change in future releases. .. warning:: ``sample_args`` for each callable must contain only Tensors. Other types are not allowed. .. warning:: Returned callables do not support higher order differentiation (e.g., double backward). .. warning:: In any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. Buffers must have ``requires_grad=False``. .. warning:: After you pass a :class:`torch.nn.Module` through :func:`~make_graphed_callables`, you may not add or remove any of that Module's parameters or buffers. .. warning:: :class:`torch.nn.Module`\s passed to :func:`~torch.cuda.make_graphed_callables` must not have module hooks registered on them at the time they are passed. However, registering hooks on modules *after* passing them through :func:`~torch.cuda.make_graphed_callables` is allowed. .. warning:: When running a graphed callable, you must pass its arguments in the same order and format they appeared in that callable's ``sample_args``. .. warning:: The automatic mixed precision is supported in :func:`~torch.cuda.make_graphed_callables` only with disabled caching. The context manager `torch.cuda.amp.autocast()` must have `cache_enabled=False`. z_make_graphed_callables does not support the autocast caching. Please set `cache_enabled=False`.FT.rzModules must not have hooks registered at the time they are passed. However, registering hooks on modules after passing them through make_graphed_callables is allowed.c3(K|] }|jduVdS)FN requires_grad.0bs r! z)make_graphed_callables..s)EEAq%/EEEEEEr zIn any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. All buffers must have ``requires_grad=False``.c3JK|]}t|tjVdSr() isinstancer#r )r}args r!rz)make_graphed_callables..s.HHS:c5<00HHHHHHr zfIn the beta API, sample_args for each callable must contain only Tensors. Other types are not allowed.c,g|]}t|Sr)len)r}rds r! z*make_graphed_callables..s!L!L!L#d))!L!L!Lr cg|]D}t|tjjr!t |ndES)r)rr#nnModuletuple parameters)r}cs r!rz*make_graphed_callables..sP""" ",Aux!?!?GallnnR"""r c2g|]}||zSrr)r}iflatten_sample_argsper_callable_module_paramss r!rz*make_graphed_callables..s9***  A!;A!>>***r cJg|] }tj!Srr#r$rr}_s r!rz*make_graphed_callables..&HHHQ%*&&((HHHr cJg|] }tj!Srrrs r!rz*make_graphed_callables..rr N)NNNc3(K|] }|j |VdSr(rzr}os r!rz)make_graphed_callables..s)$K$K11?$KQ$K$K$K$K$K$Kr c3(K|] }|j |VdSr(rzr}rs r!rz)make_graphed_callables..s=%%"#q%%%%%%%r c3LK|]}|j tj|V dSr(r{r# empty_likers r!rz)make_graphed_callables..sH++45AO+!,Q//++++++r )outputsinputs grad_outputs only_inputs allow_unused)r.c3PK|]!}|jrtj|ndV"dSr(rrs r!rz)make_graphed_callables..sJ$ $ AB1? .s)JJ1!/JQJJJJJJr c3(K|] }|j |VdSr(rzrs r!rz)make_graphed_callables..s) T TqAO T T T T T T Tr c3K|]}||V dSr(rrs r!rz)make_graphed_callables..s"&W&WQq&W&Wr  fwd_graphr bwd_graph module_paramstuple[torch.nn.Parameter, ...] len_user_argsrEoutput_unflatten_spectorch.utils._pytree.TreeSpecstatic_input_surfacernstatic_outputsstatic_grad_outputstuple[Optional[Tensor], ...]static_grad_inputsrCallable[..., object]c t Gfddtjj d fd } | S)NceZdZed fd Zeejjjd fd Z d S) Omake_graphed_callables..make_graphed_autograd_function..Graphedctxrerr rrnc^tD]Y}|||kr!|||Zt t sJt dDS)Nc3>K|]}|VdSr(detachrs r!rzjmake_graphed_callables..make_graphed_autograd_function..Graphed.forward..s*@@AQXXZZ@@@@@@r )rangedata_ptrcopy_r<rr)rrrrrrrs r!forwardzWmake_graphed_callables..make_graphed_autograd_function..Graphed.forward s}--AAA+A.7799VAY=O=O=Q=QQQ,Q/55fQi@@@  """!.%88888@@@@@@@@r gradsc~t|tksJt|D]F\}}|?||kr||Gt t sJt dDS)Nc3FK|]}||n|VdSr(rr|s r!rzkmake_graphed_callables..make_graphed_autograd_function..Graphed.backward..%sC;.make_graphed_autograd_function..Graphed.backwards5zzS)<%=%====="#6>>**GAt}::<<4==??::GGDMMM  """""4e<<<<<@Rr N)rrerr rrn)rrerr rrn) rLrMrN staticmethodrr#autogradfunctiononce_differentiabler)rrrrrrrsr!Graphedr s  A A A A A A A A\ A ^ $ 8       9 8\   r r user_argsrerctjjj|}jt |z}tjj|Sr()r#utils_pytreearg_tree_leavesapplyrtree_unflatten)rflatten_user_argsoutrrrs r!functionalizedzVmake_graphed_callables..make_graphed_autograd_function..functionalized)sP!& 3 CY O '-%(9":":]"JLC;&55c;PQQ Qr )rrerre)r#rFunction) rrrrrrrrrrrs ````````` @r!make_graphed_autograd_functionz>make_graphed_callables..make_graphed_autograd_functions             en-   : R R R R R R R Rr funcrigraph_training_statergraphedCallable[_P, _R]orig_fwdc dfd }|S) Nr_P.args user_kwargs _P.kwargsrrc:jkr|i|S|i|Sr()training)rrrrrrs r!new_fwdzEmake_graphed_callables..make_graphed_forward..new_fwdJs=}(<<<&w A[AAA'xBkBBBr )rrrrrrr)rrrrrs```` r!make_graphed_forwardz4make_graphed_callables..make_graphed_forwardDsC CCCCCCCCCr )rrrrrrrrErrrrnrrnrrrrnrr) rrirrrrrrrr)'r#is_autocast_enabledis_autocast_cache_enabled RuntimeErrorrrtypingcastr rrrr_backward_hooks_forward_hooks_forward_pre_hooksallbuffersrrrappendrrr$r\rUrW tree_leavesrrr tree_flattenreversedr{reverse enumeraterr)+rlrmrorpr.just_one_callable _sample_argsrrd flatten_argper_callable_len_user_args"per_callable_static_input_surfaces fwd_graphs bwd_graphsmempoolrr grad_inputsr outputs_gradrvper_callable_static_outputs"per_callable_output_unflatten_specr func_outputsflatten_outputsspec per_callable_static_grad_outputsper_callable_static_grad_inputsrrrrgrad_idxrrretrrrrrs+ @@r!rr%s R  "" u'F'H'H  m    i ' 'P L  E&#+$6 DDF {5vs{);S)@#A;OO y,//  4 a ) ) A%&&!++())Q..,--222]323EEEEEEE  -  E k)94@ ""5#5#5666HHKHHHHH   Z  H "M!L8K!L!L!L"""""*****s9~~&&***& IH%I2G2GHHHJHH%I2G2GHHHJ%)\!!!tG  J   5:,,.. / /03 |%G1 1    ,D$,2B .K,+,,  +-99$$+FF$$K$K$K$K$KKK |$$q(("'."5"5 ,$%%';%%%  &+++9@+++&&%)%7#6 # #K|[9  A ' . J#%)+&!$Y j!I!I88dI Z  ig  6 6 ' '4;L ' ' ' ' ' ' ' ' ' ' ' ' ' ' '!& 3 @ @ N N#**5+A+ABBB*11$7777(*$&(#;>344,--<<%C%C7ni$$ $ FT$ $ $   JJJJJJJ  |  q !!)'!::  #n11( T T,@ T T TTT!&&W&W2E&W&W&W!W!W $!3 2                 ' 0 0C  0[%<"))+h*?@@@A "))$////"#566(//0CDDD'../ABBBB%,,...#++---0000f$&CY''$ $ 400 qM qM &q ) &q ) .q 1 .q 1 ' * ,Q / +A .    dEHO , ,      0/dmWdlDL JJt     JJw    1v ::s8CN22N69N6P))P- 0P- >AVV! $V! )rrrK)rkFN) rlrjrmrnrorErprr.r/rrj) rlrsrmrtrorErprr.r/rrs) rlrvrmrwrorErprr.r/rrv)& __future__rr`rrrrrrtyping_extensionsr r r r r#r torch.cudar_utilsr__all__rrhasattr_C__dict__torch._Crrrrrrrrerjrhrrr r!rs""""""" EEEEEEEEEEEEEEAAAAAAAAAAAA (''''''     WT]]Yt__wux*++&1k,&?&?EHl#.9k:N.O.OEH*+:E+(;;EH67,,,,9999q-q-q-q-q-#q-q-q-hR(R(R(R(R(R(R(R(l %%6f8M%MNNNNN $#'   $#' (((( ($#' yyyyyyyr