0;jiO8tdZddlZddlmZddlZddlmZGddejZGddeZdS) z&Multi-Head Attention layer definition.N)Tuple)nnceZdZdZ ddedededeffd Zdej d ej d ej d e ej ej ej ffd Z ej d ejfd ej dej dej d ej fdZ ej d ejejdejdfdej d ej d ej dej dej dej d e ej ej ffdZxZS)MultiHeadedAttentionzMulti-Head Attention layer. Args: n_head (int): The number of heads. n_feat (int): The number of features. dropout_rate (float): Dropout rate. Tn_headn_feat dropout_ratekey_biasct||zdksJ||z|_||_t j|||_t j||||_t j|||_t j|||_ t j ||_ dS)z)Construct an MultiHeadedAttention object.rbias)pN) super__init__d_khrLinearlinear_qlinear_klinear_v linear_outDropoutdropoutselfrrr r __class__s g/root/voice-cloning/.venv/lib/python3.11/site-packages/chatterbox/models/s3gen/transformer/attention.pyrzMultiHeadedAttention.__init__$s !####V# &&11  &&x@@@  &&11 )FF33zL111 querykeyvaluereturnc|d}|||d|j|j}|||d|j|j}|||d|j|j}|dd}|dd}|dd}|||fS)aKTransform query, key and value. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). Returns: torch.Tensor: Transformed query tensor, size (#batch, n_head, time1, d_k). torch.Tensor: Transformed key tensor, size (#batch, n_head, time2, d_k). torch.Tensor: Transformed value tensor, size (#batch, n_head, time2, d_k). r)sizerviewrrrr transpose)rrr r!n_batchqkvs r forward_qkvz MultiHeadedAttention.forward_qkv5s&**Q-- MM% % %gr4648 D D MM#   # #GR B B MM% % %gr4648 D D KK1   KK1   KK1  !Qwrrrrdtypescoresmaskc|d}|ddkr|dd}|ddddddd|df}||t d }t j|d|d}nt j|d}||}t j||}| dd |d|j |j z}||S) a/Compute attention context vector. Args: value (torch.Tensor): Transformed value, size (#batch, n_head, time2, d_k). scores (torch.Tensor): Attention score, size (#batch, n_head, time1, time2). mask (torch.Tensor): Mask, size (#batch, 1, time2) or (#batch, time1, time2), (0, 0, 0) means fake mask. Returns: torch.Tensor: Transformed value (#batch, time1, d_model) weighted by the attention score (#batch, time1, time2). rr&r%Nr$infdimg)r' unsqueezeeq masked_fillfloattorchsoftmaxrmatmulr) contiguousr(rrr)rr!r2r3r*attnp_attnxs rforward_attentionz&MultiHeadedAttention.forward_attentionRsM***Q-- 99Q<>!$$''**D111aaa!1&++b//!112D''uU||m<r)mathsqrtrrC)rrr r!r3rErFr+r,r- key_cache value_cache new_cacher2s rforwardzMultiHeadedAttention.forwardsN""5#u551a$ ::a==1  %*[16B11D57&9&9&9 "I{ 9a.a000A ;*222AIq!f"--- aR!4!455 $(8K8KK%%a66 AArT)__name__ __module__ __qualname____doc__intr;boolrr<Tensorrr.onesrCemptyzerosrP __classcell__rs@rrrs#' 2222 %2 222222"\(- =B\ u|U\5<7 8B(UZ DDD -"-"|-" -"l -"  -"-"-"-"h(UZ DDD + A)ek,77DBDB|DB\DB| DB l DB  DB|DB u|U\) *DBDBDBDBDBDBDBDBrrcJeZdZdZ ddedededeffd Zdej d ej fd Z ej d ej ej d ej dfdej dej dej dej dej dej d eej ej ffdZxZS)RelPositionMultiHeadedAttentionzMulti-Head Attention layer with relative position encoding. Paper: https://arxiv.org/abs/1901.02860 Args: n_head (int): The number of heads. n_feat (int): The number of features. dropout_rate (float): Dropout rate. Trrr r ct||||tj||d|_tjt j|j|j |_ tjt j|j|j |_ t jj |j t jj |j dS)z4Construct an RelPositionMultiHeadedAttention object.Fr N)rrrr linear_pos Parameterr<rXrr pos_bias_u pos_bias_vinitxavier_uniform_rs rrz(RelPositionMultiHeadedAttention.__init__s x@@@)FF???,u|DFDH'E'EFF,u|DFDH'E'EFF  %%do666  %%do66666rrBr"ctj|d|d|ddf|j|j}tj||gd}||d|d|ddz|d}|ddddddf|ddddddd|ddzdzf}|S) zCompute relative positional encoding. Args: x (torch.Tensor): Input tensor (batch, head, time1, 2*time1-1). time1 means the length of query vector. Returns: torch.Tensor: Output tensor. rr%r&)devicer1r$r6N)r<r[r'rhr1rJr(view_as)rrBzero_padx_paddeds r rel_shiftz)RelPositionMultiHeadedAttention.rel_shifts#; QVVXXa[!&&((1+qI&'h%&W...9h]333==!!"!!"Qq ;; QQQ122X  & &q ) ) AAqqq!!!*qvvbzzQ** * rr/r0rrDrr r!r3rErFc|||||\}}} |dd}|ddkr`tj||ddzd\} } tj| |gd}tj| | gd} tj|| fd} |d} ||| d|j|j }|dd}||j |j zdd}||j |j zdd}tj||dd}tj||dd}|j|jkr||}||zt#j|j z }|| ||| fS)aCompute 'Scaled Dot Product Attention' with rel. positional encoding. Args: query (torch.Tensor): Query tensor (#batch, time1, size). key (torch.Tensor): Key tensor (#batch, time2, size). value (torch.Tensor): Value tensor (#batch, time2, size). mask (torch.Tensor): Mask tensor (#batch, 1, time2) or (#batch, time1, time2), (0, 0, 0) means fake mask. pos_emb (torch.Tensor): Positional embedding tensor (#batch, time2, size). cache (torch.Tensor): Cache tensor (1, head, cache_t, d_k * 2), where `cache_t == chunk_size * num_decoding_left_chunks` and `head * d_k == size` Returns: torch.Tensor: Output tensor (#batch, time1, d_model). torch.Tensor: Cache tensor (1, head, cache_t + time1, d_k * 2) where `cache_t == chunk_size * num_decoding_left_chunks` and `head * d_k == size` r%r&rr$r6rH)r.r)r'r<rIrJrar(rrrctorhrdr>shapermrKrLrC)rrr r!r3rErFr+r,r-rMrNrO n_batch_posr q_with_bias_u q_with_bias_v matrix_ac matrix_bdr2s rrPz'RelPositionMultiHeadedAttention.forwards6""5#u551a KK1  $ ::a==1  %*[16B11D57&9&9&9 "I{ 9a.a000A ;*222AIq!f"--- ll1oo OOG $ $ ) )+r4648 L L KK1  T_//999DDQJJ T_//999DDQJJ L B0C0CDD L B0C0CDD ?io - -y11Ii'49 H,,%%a66 AArrQ)rRrSrTrUrVr;rWrr<rXrmrYrZr[rrPr\r]s@rr_r_sf#' 7777 %7 777777 5<EL:(UZ DDD + A)ek,77QBQB|QB\QB| QB l QB  QB|QB u|U\) *QBQBQBQBQBQBQBQBrr_) rUrKtypingrr<rModulerr_rrrys"-,  kBkBkBkBkB29kBkBkB\BBBBBBBBBB&:BBBBBBBBBBr