%VjiXVddlZddlmZmZddlmZmZmZmZddl Z ddl m Z ddl m Z ddl mZdefdZdeeee feeee ffd eeee feeee ffdeeeeffd Zd e jd ed eeee ffdee ge fdeee d feed feeeefffdZd e jdeee d feed feeeefffdZd e jdeee d feed feeeefffdZ d5d e jdeedee deddf dZd e jdeeeefdee dee fdZ d5d e jdeedee deddf dZ d6de jdee deedee d eede jf d!Zde jfd"Zde jfd#ZGd$d%e jZ Gd&d'e jZ! d5de jd(edee!ee d fffd)Z" d5de jd(edee ee d fee d fffd*Z#d+eee d fd fdee d ffd,Z$d-ee jdee ee d fee d fffd.Z% d7d0e&e jd1eedee'fd2e j(j)fd3Z* d7d0e&e jd1eedee'fd2e j(j)fd4Z+dS)8N)IterableSequence)AnyCallableNoReturnUnion)Tensor)NamedMemberAccessorreturnc td)Na$make_functional(module): we don't yet support models that do parameter tying (also sometimes known as weight sharing). Please try to rewrite your model by replacing all instances of the tied parameter with another and/or comment your support in https://github.com/pytorch/functorch/issues/446) RuntimeErrorZ/root/voice-cloning/.venv/lib/python3.11/site-packages/torch/_functorch/make_functional.pyraise_parameter_tying_errorrs  :  r named_paramstied_named_paramsct|}t|}t|}t|}||sJi}|D] \}}|gf||< |D],\}}||vsJ||d|-t|S)a[ named_params is a dictionary of tensors: {'A': A, 'B': B} tied_named_params is another dictionary of tensors {'A': A, 'B': B, 'B_tied': B} with potentially tied (or 'duplicated') tensors This function creates a mapping from the names in named_params to the names in tied_named_params: {'A': ['A'], 'B': ['B', 'B_tied']}. )dictsetkeysissubsetitemsappendvalues)rrtensors_dict_keystied_tensors_dict_keystensor_to_mappingkeytensors rcreate_names_mapr"!s %%L.//L--//00 !2!7!7!9!9::  % %&< = === ==?#))++.. V%("I&!!(..0011 V*****&!!$++C0000 !((** + ++rmod named_members.subclassct|d}t|d}t||}i}t|}|D]I\}} | |vr"|tj| d|| <|| } ||| Jt |dkrd\} } n t|\} } | | |fS)NF)remove_duplicateTmeta)devicerrr)tupler"r torch empty_like set_tensorlenzip) r#r$r%all_named_membersunique_named_members names_mapmemoaccessornamep replacementnamesparamss r_extract_membersr;=s mmUCCCDD !E!E!EFF !57HIII D"3''H$//a D==hu/&AAABBDG1g D+....   A%% vv12 v 5) ##rcBt||jtjS)aZ This function removes all the Parameters from the model and return them as a tuple as well as their original attribute names. The weights must be re-loaded with `load_weights` before the model can be used again. Note that this function modifies the model in place and after this call, mod.parameters() will be empty. )r;named_parametersnn Parameterr#s rextract_weightsrAVs C!5r| D DDrc0t||jdS)Nc|SNr)xs rz!extract_buffers..gsar)r; named_buffersr@s rextract_buffersrHds C!2KK @ @@rFr9r: as_paramsclt|}|r d|D}|||dS)a  Reload a set of weights so that `mod` can be used again to perform a forward pass. Note that the `params` are regular Tensors (that can have history) and so are left as Tensors. This means that mod.parameters() will still be empty after this call. c6g|]}tj|Sr)r>r?).0r7s r z load_weights..ws 222a",q//222rNr set_tensors)r#r9r:rIr5s r load_weightsrPjsH#3''H3226222 '''''rr3elemsc4g}t|}t||D]c\\}}}t|D]K\}} |dkr*||| |5|| |Ld|S)Nr)r r0r enumerater swap_tensorr.) r#r3rQresultr5_ attr_nameselemi attr_names r _swap_stater[{sF"3''H!$Y__%6%6!>!>55J%j11 5 5LAyAvv h229dCCDDDD##It4444  5 MrbufferscPt|}|||dSrDrN)r#r9r\rIr5s r load_buffersr^s- #3''H (((((rrmodelweights weight_names buffer_namesct|t|ksJt|||t|dkr3t|t|ksJt||||S)zload_state(model, weights, weight_names, buffers=(), buffer_names=()) -> model load_state takes `weights` and `buffers` and assigns them to the model. This is the inverse operation of `make_functional_deprecated_v1`. r)r/rPr^)r_r`rar\rbs r load_staterds| |  G , , , , g... 7||a<  CLL0000UL'222 Lrct}t|dkrtdt \}}fd}||fS)amake_functional_deprecated_v1(model) -> weights, func, weight_names Given an nn.Module, make_functional_deprecated_v1 extracts the state (weights) and returns a functional version of the model, `func`. This makes it so that it is possible use transforms over the parameters of `model`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, func, _ = make_functional_deprecated_v1(model) func(weights, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, _, func = make_functional_deprecated_v1(model) grad_weights = grad(func)(weights, (x,)) ``` To put the state back into a model, use `load_state`. rzmake_functional_deprecated_v1(model): `model` has buffers. Please use make_functional_with_buffers_deprecated_v1(model) instead.cXtj}t||||SrD)copydeepcopyrP)r`data mutable_model descriptorsr_s rfunz*make_functional_deprecated_v1..funs1 e,, ]K999}d##r)listr\r/r rA)r_r\r`rVrlrks` @rmake_functional_deprecated_v1rns45==??##G 7||a I   .e44G[!$$$$$$ C $$rcpt\}}t\}}fd}|||fS)a`make_functional_with_buffers_deprecated_v1(model) -> weights, buffers, func, weight_names, buffer_names Given an nn.Module, make_functional_with_buffers_deprecated_v1 extracts the state (weights and buffers) and returns a functional version of the model, `func`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) grad_weights = grad(func)(weights, buffers, (x,)) ``` To put the state back into a model, use `load_state`. cztj}t||t||||SrD)rgrhrPr^)r`r\rirjbuf_descriptorsr_weight_descriptorss rrlz7make_functional_with_buffers_deprecated_v1..funsD e,, ]$6@@@]OW===}d##r)rArH)r_r`rVr\rlrqrrs` @@r*make_functional_with_buffers_deprecated_v1rssg2&5U%;%;"G "1%"8"8G_a$$$$$$$ GS"4o EErc .eZdZdZdejdeedfdeedfdeee efdeee efdd f fd Z e dd ejd e dedee dfee dfffdZdee dee defdZxZS)FunctionalModuleWithBufferszW This is the callable object returned by :func:`make_functional_with_buffers`. stateless_model param_names.rbparam_names_mapbuffer_names_mapr Nct||_||_||_t ||_|j|dSrD)super__init__rvrwrbr all_names_mapupdate)selfrvrwrbrxry __class__s rr|z$FunctionalModuleWithBuffers.__init__sa .&(!/22 !!"233333rFr_disable_autograd_trackingctj|}t|\}}}t|\}}}|r|D]} | dt |||||||fSNF)rgrhrArHrequires_grad_ru) r_r model_copyr:rwrxr\rbryparams r _create_fromz(FunctionalModuleWithBuffers._create_from s ]5)) />z/J/J, _2A*2M2M// $ , , ,$$U++++ 'KHX       rr:r\ct|j|jt|t|z} |j|i|t|j|j|S#t|j|j|wxYwrD)r[rvr}r+)rr:r\argskwargs old_states rforwardz#FunctionalModuleWithBuffers.forwards    &MME'NN *   M'4'888 ,d.@) L L L LK,d.@) L L L Ls A""A?F__name__ __module__ __qualname____doc__r>Moduler+strrrmr| staticmethodboolr rrrr __classcell__rs@rrurusP4438_4CHo 4 c49n- 4 sDI~. 4 444444 sl ]5)) )8)D)D& Y $ , , ,$$U++++ KCCVKKrr:ct|j|j|} |j|i|t|j|j|S#t|j|j|wxYwrD)r[rvr3)rr:rrrs rrzFunctionalModule.forwardJsl 4dnfMM  I'4'888 ,dni H H H HK,dni H H H Hs AA"rrrs@rrr.s # #38_ #T#Y' #  # # # # # # func, params Given a ``torch.nn.Module``, :func:`make_functional` extracts the state (params) and returns a functional version of the model, ``func``. This makes it so that it is possible use transforms over the parameters of ``model``. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) func(params, x) And here is an example of applying the grad transform over the parameters of a model. .. code-block:: python import torch import torch.nn as nn from functorch import make_functional, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) def compute_loss(params, x, t): y = func(params, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, x, t) If the model has any buffers, please use :func:`make_functional_with_buffers` instead. Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. rzdmake_functional(model): `model` has buffers. Please use make_functional_with_buffers(model) instead.r)rmr\r/r rr)r_rr\s rmake_functionalrTsgz5==??##G 7||a ;     ( ( )B )  rc:t||S)amake_functional_with_buffers(model, disable_autograd_tracking=False) -> func, params, buffers Given a ``torch.nn.Module``, make_functional_with_buffers extracts the state (params and buffers) and returns a functional version of the model ``func`` that can be invoked like a function. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) func(params, buffers, x) And here is an example of applying the grad transform over the parameters of a model: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) def compute_loss(params, buffers, x, t): y = func(params, buffers, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, buffers, x, t) Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. r)rur)r_rs rmake_functional_with_buffersrs(t ' 3 3 )B 4  rtuple_of_tuple_of_tensorscdtt|}td|D}|S)Nc3bK|]*}tj|V+dSrDr,stackdetachrLshardss r z"transpose_stack..sI)/ F""$$r)r+r0)rresultss rtranspose_stackrsJ!&c+D&E F F3LG Nrmodelsct|dkrtdtd|Ds(td|Dstdt|dtfd|Dstdt d|D\}}}t |}t |}|d||fS) a(combine_state_for_ensemble(models) -> func, params, buffers Prepares a list of torch.nn.Modules for ensembling with :func:`vmap`. Given a list of ``M`` ``nn.Modules`` of the same class, stacks all of their parameters and buffers together to make ``params`` and ``buffers``. Each parameter and buffer in the result will have an additional dimension of size ``M``. :func:`combine_state_for_ensemble` also returns ``func``, a functional version of one of the models in :attr:`models`. One cannot directly run ``func(params, buffers, *args, **kwargs)`` directly, you probably want to use ``vmap(func, ...)(params, buffers, *args, **kwargs)`` Here's an example of how to ensemble over a very simple model: .. code-block:: python num_models = 5 batch_size = 64 in_features, out_features = 3, 3 models = [torch.nn.Linear(in_features, out_features) for i in range(num_models)] data = torch.randn(batch_size, 3) fmodel, params, buffers = combine_state_for_ensemble(models) output = vmap(fmodel, (0, 0, None))(params, buffers, data) assert output.shape == (num_models, batch_size, out_features) .. warning:: All of the modules being stacked together must be the same (except for the values of their parameters/buffers). For example, they should be in the same mode (training vs eval). This API is subject to change -- we're investigating better ways to create ensembles and would love your feedback how to improve this. rz?combine_state_for_ensemble: Expected at least one model, got 0.c3$K|] }|jV dSrDtrainingrLms rrz-combine_state_for_ensemble..s$++q ++++++rc3&K|] }|j V dSrDrrs rrz-combine_state_for_ensemble..s&2R2Raqz>2R2R2R2R2R2RrzTcombine_state_for_ensemble: Expected all models to have the same training/eval mode.c3>K|]}t|kVdSrD)type)rLr model0_typs rrz-combine_state_for_ensemble..s.55tAww*$555555rzHcombine_state_for_ensemble: Expected all models to be of the same class.c,g|]}t|Sr)rrLr_s rrMz.combine_state_for_ensemble..s! B B B% &u - - B B Br)r/r allrr0r)rfuncsr:r\rs @rcombine_state_for_ensemblersP 6{{a M    ++F+++ + + s2R2R62R2R2R/R/R  0   fQiJ 5555f555 5 5  V   ! B B6 B B BE67V $ $Fg&&G 8VW $$rcpu model_classensemble_shaper)cfd}|S)Nc*t dkrtdt dkr* i }t|S d}|dkrtd|dt  fdt |D}t i\}}}t d|D}t t |}t d|D}|||fS) N,NYI: ensemble_shape with more than 1 elementr num_models  should be > 0c3NK|]}iV dSrDtorLrVrr)rrs rrz3functional_init..wrapped..2Q  89KK ( ( ( + +F 3 3      rc3@K|]}t|dVdS)rN)rnrs rrz3functional_init..wrapped..6s0TTE5e<.wrapped..88KK F++2244KKKKKKr)r/ ValueErrorrrnr+ranger0) rrr_ num_modelsrrVfnr9r`r)rrs `` rwrappedz functional_init..wrapped(s] ~  ! # #KLL L ~  ! # #K00033F;;E077 7#A& ??E:EEEFF F       =B:=N=N     5[[$5Q&5Q5QRR 2uTTVTTTTTW &&KK7KKKKKE!!rrrrr)rs``` rfunctional_initr#s/ """""""& Nrcfd}|S)Nct dkrtdt dkr* i }t|S d}|dkrtd|dt  fdt |D}t  i\}}}}}tt d|D\} } t t| } t d| D} t t| } t d | D} | | |||fS) Nrrrrrc3NK|]}iV dSrDrrs rrz@functional_init_with_buffers..wrapped..Mrrc3DK|]}t|ddVdS)Nr)rsrs rrz@functional_init_with_buffers..wrapped..XsG;5AA"1"Erc3bK|]*}tj|V+dSrDrrs rrz@functional_init_with_buffers..wrapped..^rrc3bK|]*}tj|V+dSrDrrs rrz@functional_init_with_buffers..wrapped..`rr)r/rrrnr+rrsr0)rrr_rrrVrrarbr`r\r)rrs`` rrz-functional_init_with_buffers..wrappedCs ~  ! # #KLL L ~  ! # #K00033F;;E077 7#A& ??E:EEEFF F       =B:=N=N      7{{D7SF7S7S T T     #  W &&KK7KKKKKW &&KK7KKKKK\<??rrrs``` rfunctional_init_with_buffersr>s7 @@@@@@@@ Nrrr*)rr),rgcollections.abcrrtypingrrrrr,torch.nnr>r %torch.nn.utils._named_member_accessorr rrrr+rmr"rr;rArHrrPr[r^rdrnrsrurrrrrrinttypesDevicerrrrrrs] ........111111111111 EEEEEEX,S&[)8E#v+4F+GGH,T#v+.sF{9K0LLM, #tCy.,,,,8$ $C%V *$x'($ 5 uS#XS$s)^0D DE $$$$2 E  E 5 uS#XS$s)^0D DE E E E EA A 5 uS#XS$s)^0D DEAAAA (( ( C=( V ( (  (((("   #CcN3 EE 9E15E U63;/ /0EEEER9><< 9<15< &fck(:E&#+