~VjiV\ddlZddlmZmZddlZddlmZddlmZgZ dd ejd ejd ejd e d e de de dejfdZ GddejjZGddejjZGddejjZGddejjZdS)N)OptionalUnion)Tensor) functional ref_channelTHz>:0yE>psd_spsd_nreference_vectorsolutiondiagonal_loadingdiag_epsepsreturnc|dkrtj||||||}nM|dkrtj|}ntj|||||}tj||||||}|S)aCompute the MVDR beamforming weights with ``solution`` argument. Args: psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. reference_vector (torch.Tensor): one-hot reference channel matrix. solution (str, optional): Solution to compute the MVDR beamforming weights. Options: [``ref_channel``, ``stv_evd``, ``stv_power``]. (Default: ``ref_channel``) diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: the mvdr beamforming weight matrix rstv_evd)rr)Fmvdr_weights_soudenrtf_evd rtf_powermvdr_weights_rtf) r r r r rrrbeamform_vectorstvs ^/root/voice-cloning/.venv/lib/python3.11/site-packages/torchaudio/transforms/_multi_channel.py_get_mvdr_vectorrs:=  /u>NP`bjlopp y )E""CC+eU,rr0r8zerosregister_buffer) r'rr r!r=rr>r r rBrCr(s rr&z MVDR.__init__s      ]ddemnn '  $(   z??#k!nn#k!nn#(;q>> #(;q>>  We,,, We,,, \:666 \:66666rr r r mask_smask_nr rrrc d|jr,|d}|d}|jjdkrY||_||_|d|_|d|_t||||||| S| ||}| ||}||_||_|j|dz|_|j|dz|_t||||||| S)aRecursively update the MVDR beamforming vector. Args: psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. mask_s (torch.Tensor): Time-Frequency mask of the target speech. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. mask_n (torch.Tensor or None, optional): Time-Frequency mask of the noise. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. reference_vector (torch.Tensor): One-hot reference channel matrix. solution (str, optional): Solution to compute the MVDR beamforming weights. Options: [``ref_channel``, ``stv_evd``, ``stv_power``]. (Default: ``ref_channel``) diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: The MVDR beamforming weight matrix. r,r-rA) r!r/r ndimr sumrBrCr_get_updated_psd_speech_get_updated_psd_noise) r'r r rHrIr r rrrs r_get_updated_mvdr_vectorzMVDR._get_updated_mvdr_vectors(L ? )[[R[((F[[R[((F :?a  DJDJ$jjRj00DO$jjRj00DO#E52BHN^`hjmnn n00??E//v>>EDJDJ"o r 0B0BBDO"o r 0B0BBDO#E52BHN^`hjmnn nrc|j|j|dzz }d|j|dzz }|j|dz||dzz}|S)aUpdate psd of speech recursively. Args: psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. mask_s (torch.Tensor): Time-Frequency mask of the target speech. Tensor with dimensions `(..., freq, time)`. Returns: torch.Tensor: The updated PSD matrix of target speech. rKr-rA.NN)rBrMr )r'r rH numerator denominators rrNzMVDR._get_updated_psd_speechiOt9K9K'KL 4?VZZBZ-?-??@  Y77%+oB^:^^ rc|j|j|dzz }d|j|dzz }|j|dz||dzz}|S)aUpdate psd of noise recursively. Args: psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. mask_n (torch.Tensor or None, optional): Time-Frequency mask of the noise. Tensor with dimensions `(..., freq, time)`. Returns: torch.Tensor: The updated PSD matrix of noise. rKr-rArR)rCrMr )r'r rIrSrTs rrOzMVDR._get_updated_psd_noiserUrNr)c N|j}|jdkrtd|j|std|j|jt jkr|}|tj dd|z }| ||}| ||}t j | dd|j t j}|d |jfd|jr,|||||||j|j|j}n#t+||||j|j|j}t-j||} | |S) a`Perform MVDR beamforming. Args: specgram (torch.Tensor): Multi-channel complex-valued spectrum. Tensor with dimensions `(..., channel, freq, time)` mask_s (torch.Tensor): Time-Frequency mask of target speech. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. mask_n (torch.Tensor or None, optional): Time-Frequency mask of noise. Tensor with dimensions `(..., freq, time)` if multi_mask is ``False`` or with dimensions `(..., channel, freq, time)` if multi_mask is ``True``. (Default: None) Returns: torch.Tensor: Single-channel complex-valued enhanced spectrum with dimensions `(..., freq, time)`. z?Expected at least 3D tensor (..., channel, freq, time). Found: ziThe type of ``specgram`` tensor must be ``torch.cfloat`` or ``torch.cdouble``. Found: Nz=``mask_n`` is not provided, use ``1 - mask_s`` as ``mask_n``.rA)devicedtype.)r[rLrDshape is_complexr8cfloatcdoublewarningswarnr0rFsizerZrfill_r>rPr r=rrrapply_beamformingto) r'r)rHrIr[r r uw_mvdrspecgram_enhanceds rr1z MVDR.forward#s& =1  o_g_moopp p""$$ -$N--  >U\ ) )''))H > MY Z Z ZZF6**6** K ,X_EM Z Z Z #t  &&q))) ; h22uffa@QSWS`FF&eUAt}dFWY]YfggF/AA ##E***r)rrFTrFrTrr r$)r2r3r4r5intstrr6r7r&r8rrPrNrOrr1r9r:s@rr<r<asAAJ% !!7!7!7!7 !7  !7  !7!7!7!7!7!7!7T&!%7o7o|7o|7o 7o  7o  , 7o7o7o7o7o 7o7o7o7orU\5<TYT`"EL%,SXS_$^b1+1+ 1+.3l1+DLU\DZ1+ 1+1+1+1+1+1+1+1+rr<cReZdZdZ ddedededeeefd ed ed ed efd Z dS)RTFMVDRaMinimum Variance Distortionless Response (*MVDR* :cite:`capon1969high`) module based on the relative transfer function (RTF) and power spectral density (PSD) matrix of noise. .. devices:: CPU CUDA .. properties:: Autograd TorchScript Given the multi-channel complex-valued spectrum :math:`\textbf{Y}`, the relative transfer function (RTF) matrix or the steering vector of target speech :math:`\bm{v}`, the PSD matrix of noise :math:`\bf{\Phi}_{\textbf{NN}}`, and a one-hot vector that represents the reference channel :math:`\bf{u}`, the module computes the single-channel complex-valued spectrum of the enhanced speech :math:`\hat{\textbf{S}}`. The formula is defined as: .. math:: \hat{\textbf{S}}(f) = \textbf{w}_{\text{bf}}(f)^{\mathsf{H}} \textbf{Y}(f) where :math:`\textbf{w}_{\text{bf}}(f)` is the MVDR beamforming weight for the :math:`f`-th frequency bin, :math:`(.)^{\mathsf{H}}` denotes the Hermitian Conjugate operation. The beamforming weight is computed by: .. math:: \textbf{w}_{\text{MVDR}}(f) = \frac{{{\bf{\Phi}_{\textbf{NN}}^{-1}}(f){\bm{v}}(f)}} {{\bm{v}^{\mathsf{H}}}(f){\bf{\Phi}_{\textbf{NN}}^{-1}}(f){\bm{v}}(f)} Trr r)rtfr reference_channelrrrrcbtj||||||}tj||} | S)a Args: specgram (torch.Tensor): Multi-channel complex-valued spectrum. Tensor with dimensions `(..., channel, freq, time)` rtf (torch.Tensor): The complex-valued RTF vector of target speech. Tensor with dimensions `(..., freq, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. reference_channel (int or torch.Tensor): Specifies the reference channel. If the dtype is ``int``, it represents the reference channel index. If the dtype is ``torch.Tensor``, its shape is `(..., channel)`, where the ``channel`` dimension is one-hot. diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: Single-channel complex-valued enhanced spectrum with dimensions `(..., freq, time)`. )rrrd) r'r)rnr rorrrrgspectrum_enhanceds rr1zRTFMVDR.forwardrs<@#C0ACSU]_bcc/AA  rNTrr ) r2r3r4r5rrrjr6r7r1rrrmrmWs@"&"!"!"!"! "! !f- "!  "!"!"! "!"!"!"!"!"!rrmc\eZdZdZ ddedededeeefd ed ed ed e jfd Z dS) SoudenMVDRaMinimum Variance Distortionless Response (*MVDR* :cite:`capon1969high`) module based on the method proposed by *Souden et, al.* :cite:`souden2009optimal`. .. devices:: CPU CUDA .. properties:: Autograd TorchScript Given the multi-channel complex-valued spectrum :math:`\textbf{Y}`, the power spectral density (PSD) matrix of target speech :math:`\bf{\Phi}_{\textbf{SS}}`, the PSD matrix of noise :math:`\bf{\Phi}_{\textbf{NN}}`, and a one-hot vector that represents the reference channel :math:`\bf{u}`, the module computes the single-channel complex-valued spectrum of the enhanced speech :math:`\hat{\textbf{S}}`. The formula is defined as: .. math:: \hat{\textbf{S}}(f) = \textbf{w}_{\text{bf}}(f)^{\mathsf{H}} \textbf{Y}(f) where :math:`\textbf{w}_{\text{bf}}(f)` is the MVDR beamforming weight for the :math:`f`-th frequency bin. The beamforming weight is computed by: .. math:: \textbf{w}_{\text{MVDR}}(f) = \frac{{{\bf{\Phi}_{\textbf{NN}}^{-1}}(f){\bf{\Phi}_{\textbf{SS}}}}(f)} {\text{Trace}({{{\bf{\Phi}_{\textbf{NN}}^{-1}}(f) \bf{\Phi}_{\textbf{SS}}}(f))}}\bm{u} Trr r)r r rorrrrcbtj||||||}tj||} | S)a Args: specgram (torch.Tensor): Multi-channel complex-valued spectrum. Tensor with dimensions `(..., channel, freq, time)`. psd_s (torch.Tensor): The complex-valued power spectral density (PSD) matrix of target speech. Tensor with dimensions `(..., freq, channel, channel)`. psd_n (torch.Tensor): The complex-valued power spectral density (PSD) matrix of noise. Tensor with dimensions `(..., freq, channel, channel)`. reference_channel (int or torch.Tensor): Specifies the reference channel. If the dtype is ``int``, it represents the reference channel index. If the dtype is ``torch.Tensor``, its shape is `(..., channel)`, where the ``channel`` dimension is one-hot. diagonal_loading (bool, optional): If ``True``, enables applying diagonal loading to ``psd_n``. (Default: ``True``) diag_eps (float, optional): The coefficient multiplied to the identity matrix for diagonal loading. It is only effective when ``diagonal_loading`` is set to ``True``. (Default: ``1e-7``) eps (float, optional): Value to add to the denominator in the beamforming weight formula. (Default: ``1e-8``) Returns: torch.Tensor: Single-channel complex-valued enhanced spectrum with dimensions `(..., freq, time)`. )rrrd) r'r)r r rorrrrgrqs rr1zSoudenMVDR.forwards<@&ue5FHXZbdghh/AA  rNrr) r2r3r4r5rrrjr6r7r8r1rsrrrurus>"&"!"!"!"! "! !f- "!  "!"!"! "!"!"!"!"!"!rruri)r`typingrrr8r torchaudiorr__all__rkr6r7rnnModulerr<rmrursrrr|s"""""""" &&&&&& "!&& <& <&l& &  &  & & \&&&&R'''''%(/'''Ts+s+s+s+s+58?s+s+s+l=!=!=!=!=!eho=!=!=!@<!<!<!<!<!<!<!<!<!<!r