QVjig&dZddlmZmZmZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZddgZGd deZd d ed e d ed ed e d ze_deedeedeedeedeedeedededededededededededef"d Zdeedeedeedeedeedeedededededededededededef"d!Ze e" d&deedeedeedeedeedeeded$eededededededededededef$d%ZdS)'z'Implementation for the NAdam algorithm.)castOptionalUnionN)Tensor)_capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _get_value _maximize_doc _params_doc_stack_if_compiling _to_scalar_use_grad_for_differentiable _view_as_real OptimizerParamsTNAdamnadamceZdZ ddddddd ed eeefd eeefd ed ededede edededeffdZ fdZ dZ e ddZxZS)rMb`?g?g+?:0yE>rMbp?FN)foreachmaximize capturabledifferentiableparamslrbetaseps weight_decaymomentum_decaydecoupled_weight_decayrr r!r"c Rt|tr'|dkrtdd|kstd|d|kstd|d|dcxkrdksntd|dd|dcxkrdksntd |dd|kstd |d|kstd |||||||| || | d } t || dS) NrzTensor lr must be 1-elementzInvalid learning rate: zInvalid epsilon value: r?z#Invalid beta parameter at index 0: z#Invalid beta parameter at index 1: zInvalid weight_decay value: zInvalid momentum_decay value: ) r$r%r&r'r(r)r rr!r") isinstancernumel ValueErrorsuper__init__)selfr#r$r%r&r'r(r)rr r!r"defaults __class__s K/root/voice-cloning/.venv/lib/python3.11/site-packages/torch/optim/nadam.pyr1zNAdam.__init__!s b& ! ! zNAdam.__setstate__Ls U###&  E   Z / / /   Y - - -   \5 1 1 1   -u 5 5 5   5u = = =8_  *..B//w<<1$$ ?76?;;#(#9#9 %\2SEL (0A0C0CAH"'h>O>Q>Q!R!R!R  !?7<+@AA&-l&; %\2VEL +3D3F3Fqx"'kARATAT!U!U!U  -    r6cd}|dD]} | j|tj| z}|| | jjrt d|| j|j| } t| dkr|dr(tjdt| j n!tj dt | d <|dr(tj dt| j n!tj d t | d <tj | tj | d<tj | tj | d<|| d|| d|| d || d |S)NFr#z'NAdam does not support sparse gradientsrr!r9r+r<r8r,r=) memory_formatexp_avg exp_avg_sq)gradrD is_complexappend is_sparse RuntimeErrorrArCzerosrr;rGones zeros_likepreserve_format) r2rHparams_with_gradgradsexp_avgs exp_avg_sqs mu_products state_steps has_complexrIrAs r5 _init_groupzNAdam._init_groupjs x$ 2$ 2Av!u/222  ''***6#R&'PQQQ QV$$$ 1 u::?? !.J B.?.A.A!(SSSS"\#5F5H5HIII&M!.J 2->-@-@RRRR"\#5F5H5HIII,' (-'7)>(((E)$+0*:)>+++E,'i 0111""5#6777""5#6777""5=111r6c|d}|5tj5|}dddn #1swxYwY|jD]}g}g}g}g}g}g} t t t t f|d\} } |||||||| } t|||||| | | |d|d|d|d|d|d|d |d |d | |S) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nr%r$r'r(r&r r)rr!r") beta1beta2r$r'r(r&r r)rr!r"ra) _cuda_graph_capture_health_checkrD enable_gradr?rtuplerFrbr) r2closurelossrHr[r\r]r^r_r`rdreras r5r8z NAdam.steps --///  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&& & E-/ "$E%'H(*K(*K(*KeUl 3U7^DDLE5** K  ;">2$%56%Lz*',-E'Fi( .$%56'%     * s AA A)rrrrrFN)__name__ __module__ __qualname__rrrFrrhboolrr1r>rbrr8 __classcell__)r4s@r5rr s;$(%1 $',)+#' $)+)+)+)+ %- )+UE\" )+  )+  )+)+!%)+$)+)+)+)+)+)+)+)+)+V<000d"666"!66666r6a Implements NAdam algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma_t \text{ (lr)}, \: \beta_1,\beta_2 \text{ (betas)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \: \lambda \text{ (weight decay)}, \:\psi \text{ (momentum decay)} \\ &\hspace{13mm} \: \textit{decoupled\_weight\_decay}, \:\textit{maximize} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0 \leftarrow 0 \text{ ( second moment)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} \\ &\hspace{5mm} \textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm}\textbf{if} \: \textit{decoupled\_weight\_decay} \\ &\hspace{15mm} \theta_t \leftarrow \theta_{t-1} - \gamma \lambda \theta_{t-1} \\ &\hspace{10mm}\textbf{else} \\ &\hspace{15mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm} \mu_t \leftarrow \beta_1 \big(1 - \frac{1}{2} 0.96^{t \psi} \big) \\ &\hspace{5mm} \mu_{t+1} \leftarrow \beta_1 \big(1 - \frac{1}{2} 0.96^{(t+1)\psi}\big)\\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow \mu_{t+1} m_t/(1-\prod_{i=1}^{t+1}\mu_i)\\[-1.ex] & \hspace{11mm} + (1-\mu_t) g_t /(1-\prod_{i=1}^{t} \mu_{i}) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\theta_t \leftarrow \theta_t - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Incorporating Nesterov Momentum into Adam`_. z Args: a lr (float, Tensor, optional): learning rate (default: 2e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) momentum_decay (float, optional): momentum momentum_decay (default: 4e-3) decoupled_weight_decay (bool, optional): whether to decouple the weight decay as in AdamW to obtain NAdamW. If True, the algorithm does not accumulate weight decay in the momentum nor variance. (default: False) z z .. _Incorporating Nesterov Momentum into Adam: https://openreview.net/forum?id=OM0jvwB8jIp57ZJjtNEZ .. _Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101 r#r\r]r^r_r`rdrer$r'r(r&r)r r!r"rac tjst|}t |D]\}}| s||n|| }||}||}||}||}tj|rPtj|}tj|}tj|}tj|}tjsZ|rXt}|j j |j j cxkr|j j krnn|j j |vsJd|d|dz }|r|}nt|}d||zz }| dkr5| r| d|| zz n||| }|ddd|| zzzz z}|ddd|dz| zzzz z}||z}||d|z | |||d|z ||}|s|ri|| }||z}|| d|z zd|z z z}|| |zd|z z z}||||||vt||z}|| |||| d|z zdt|z z |||t+t,| |zd|z z  dS) NzVIf capturable=True, params, mu_products and state_steps must be on supported devices: .rralphar,?Q?)value)rDjit is_scriptingr enumeraterS view_as_realcompiler is_compilingr r;typermul_addlerp_addcmul_divsqrtaddcdiv_add_rrF)r#r\r]r^r_r`rdrer$r'r(r&r)r r!r"raiparamrRrPrQr=step_tcapturable_supported_devicesr8bias_correction2mumu_nextdenommu_product_nexts r5_single_tensor_nadamrs( 9 ! ! # # ^^f%%II5'6uQxxeAhY1+ ^  ^ Q  E " " 8&u--E%d++D(11G+J77J~**,,  +L+N+N ( !Z%6%;QQQQv}?QQQQQQL%)EEEEI)EIIIFEF !   &DDf%%Dud{? 1  % ; 1rL001111xx\x::cC4D>,A#BCC D3$(n1L(M!NNO b   dAI&&&''d!e)'DDD/005577  Z IIcNNE)72OB3#(+sZ/?@AD"w#2G!HIG NN4 ' ' ' NN7E * * * *(44w>O JJsOOO NNeRC38$4j>T>T8T$U     NN5B3=S?5J"KLL     KIIr6c   (t|dkrdS|r JdtjsJ|rHt d(t (fdt |||DsJd(dttj ||||||g}| D]1\\}}}}}}}ttt|}ttt|}ttt|}ttt|}ttt|}ttt|}|rt||||| rtj|}tjs9|djr,tj|tjdd d ntj|d | dkrO| rtj|d | zz n1| rtj||| ntj||| }tj||d z tj|tj|||d z tj|}|r&tj| } tjd | }!tj|!dtj|!dtj|!tj| tjd | }"tj|"dtj|"dtj|"~ tj|}#tj|#dtj|#tj|#n,fd|D}# fd|D}! fd|D}"tj||!tj||#tj|| ~#|rtj|!dtj|!tj|d}$tj|$tj|!|$|!}%~$tj||"}$tj|"tj|$dtj|"|$|"}&~$tj|%|}'tj|'|&|tj||'|tAfdt ||!D}%tAfdt ||"D}&tj||||%tj||||&3dS)Nrz#_foreach ops don't support autogradF) supports_xlac3K|]D\}}}|jj|jjcxko|jjknco |jjvVEdSrk)r;r~).0rImpr8rs r5 z&_multi_tensor_nadam..sz  2t HMRY^ ? ? ? ?t{/? ? ? ? ? > !==      r6zWIf capturable=True, params, mu_products, and state_steps must be on supported devices: rrr,cpu)r;rsrrvgc@g|]}dt|zz dzS)rrur)rr8res r5 z'_multi_tensor_nadam..s=$$$;?Uj....36$$$r6c Lg|] }dddt|zzzz z!S)r,rurvrrr8rdr(s r5rz'_multi_tensor_nadam..sJsdz$/?/?./P&QRRSr6c Rg|]#}dddt|dzzzzz z$S)r,rurvrrrs r5rz'_multi_tensor_nadam..sU*T*:*:Q*>.)P QRRTr6clg|]0\}}td|z zdt|z z dz1Sr,r)rr=rr$s r5rz'_multi_tensor_nadam..8sT& B ^^sRx0C*Z:P:P4PQUWWr6clg|]0\}}t|zdt||zz z dz1Srr)rr=rr$s r5rz'_multi_tensor_nadam..>sb, G #2!"J!7!7'!AAC r6)!rCrDr|r}r allziprr"_group_tensors_by_device_and_dtypevaluesrlistrr _foreach_negis_cpu _foreach_add_rG _foreach_mul_ _foreach_add_foreach_lerp__foreach_addcmul_ _foreach_sqrt _foreach_mul _foreach_pow _foreach_sub_ _foreach_neg__foreach_sqrt_ _foreach_div_ _foreach_sub_foreach_addcdiv_r))r#r\r]r^r_r`rdrer$r'r(r&r)r r!r"ragrouped_tensorsgrouped_params_grouped_grads_grouped_exp_avgs_grouped_exp_avg_sqs_grouped_mu_products_grouped_state_steps__grouped_params grouped_gradsgrouped_exp_avgsgrouped_exp_avg_sqsgrouped_mu_productsgrouped_state_stepsexp_avg_sq_sqrtexponentmusmu_nextsbias_correction_sqrtrstep_size_gradsstep_size_expavg numeratorrs) ``` ` @r5_multi_tensor_nadamr|s*( 6{{aDDDDD  > & & ( (  Z  'H( ( ( $     #6; DD        /+ / / /     BBB +{KHO  " " $ $kk  d6lO<<T&\>:: V .?@@"4<1EFF"4<1EFF"4<1EFF    /?AT     >!.}==M ~**,, 81DQ1G1N 8  #U\#e%D%D%DC        3Q 7 7 7 1  % #NA\8I4IJJJJ'%~\%*$6%~\%%%M -}a%iHHH /777   q5y    -.ABB # )*=~NNH$T844C  T * * *  S ) ) )  U + + +  . 9 9 9)$99H  $ / / /  # . . .  % 0 0 0#(#5e=P#Q#Q   4c : : :   4 5 5 5  !5 6 6 6 6$$$$CV$$$ /C0H /555 O-ABBB OS111 ! >   S ) ) )  R ( ( (&':C@@E   & & &  U + + +!O&':HEEE  " - - -  s + + +  % 0 0 0' *?MJJI  #I/?AQ R R R  #NI O O O O1*-.A3*G*GO 3033F/Q/Q     #      #      Mkkr6)single_tensor_fnFrc td|Dstdtd|Dstd|t|| d\}}|r-tjrtd|r&tjst }nt}|||||||| | |||| |||| | dS) zpFunctional API that performs NAdam algorithm computation. See :class:`~torch.optim.NAdam` for details. c3JK|]}t|tjVdSrkr-rDrrts r5rznadam..s.@@qz!U\**@@@@@@r6zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsc3JK|]}t|tjVdSrkrrs r5rznadam..xrr6zPAPI has changed, `mu_products` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizers) rdrer$r'r(r r)r&r!r"ra)rrVr rDrxryrr)r#r\r]r^r_r`r)rr!r"rar rdrer$r'r(r&rfuncs r5rrWsC8 @@K@@@ @ @  ^    @@K@@@ @ @  ^   1 Ne   7U59))++USTTT$uy--//$"#D  !%5 %#r6)FNFFFF) __doc__typingrrrrDr optimizerrr r r r r rrrrrrrrrr__all__rrrFrorrrrNr6r5rs..(((((((((( ( G sssssIsssn&N     !O> F` L` <`6l`f ` f ` f ` ` ` ``` `!`` !`"#`$%````FX LX <X6lXf X f X f X X X XXX X!XX !X"#X$%XXXXv 1EFFF$)" DD LD <D6lDf D f D f D!Dd^DDDDD !D" #D$ %D&'D()D* +DDDGFDDDr6