0;ji%zdZddlmZmZddlZddlmZGddejZGddejZdS) z(Encoder self-attention layer definition.)OptionalTupleN)nncjeZdZdZ ddedejjdejjdede f fd Z ej d ej ej d ej d fd ej d ej dej dej dej dej deej ej ej ej ffdZxZS)TransformerEncoderLayera\Encoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` or `RelPositionMultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward`, instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): True: use layer_norm before each sub-block. False: to use layer_norm after each sub-block. Tsize self_attn feed_forward dropout_ratenormalize_beforect||_||_t j|d|_t j|d|_t j||_ ||_ ||_ dS)!Construct an EncoderLayer object.-q=epsN) super__init__r r r LayerNormnorm1norm2Dropoutdropoutrr )selfrr r r r __class__s k/root/voice-cloning/.venv/lib/python3.11/site-packages/chatterbox/models/s3gen/transformer/encoder_layer.pyrz TransformerEncoderLayer.__init__(s| "(\$E222 \$E222 z,//  0rrrdtyperrrrxmaskpos_embmask_pad att_cache cnn_cachereturnc|}|jr||}|||||||\}} |||z}|js||}|}|jr||}||||z}|js||}t jd|j|j } ||| | fS)uhCompute encoded features. Args: x (torch.Tensor): (#batch, time, size) mask (torch.Tensor): Mask tensor for the input (#batch, time，time), (0, 0, 0) means fake mask. pos_emb (torch.Tensor): just for interface compatibility to ConformerEncoderLayer mask_pad (torch.Tensor): does not used in transformer layer, just for unified api with conformer. att_cache (torch.Tensor): Cache tensor of the KEY & VALUE (#batch=1, head, cache_t1, d_k * 2), head * d_k == size. cnn_cache (torch.Tensor): Convolution cache in conformer layer (#batch=1, size, cache_t2), not used here, it's for interface compatibility to ConformerEncoderLayer. Returns: torch.Tensor: Output tensor (#batch, time, size). torch.Tensor: Mask tensor (#batch, time, time). torch.Tensor: att_cache tensor, (#batch=1, head, cache_t1 + time, d_k * 2). torch.Tensor: cnn_cahce tensor (#batch=1, size, cache_t2). )r#cacherrdevice) r rr rrr torchzerosrr+) rr!r"r#r$r%r&residualx_att new_att_cachefake_cnn_caches rforwardzTransformerEncoderLayer.forward:s@    1 A#~~aAtWT]~^^} t||E** *$  1 A    1 A t||D$5$5a$8$899 9$  1 AYagahOOO$ ~55r)T)__name__ __module__ __qualname____doc__intr,rModulefloatboolronesr-Tensorrr2 __classcell__rs@rrrs>  *"& 1118?1ho 1  1  111111.",IUZ!H!H!H"-%+l";";"-%+l";";0606 <06l06 06 , 06 < 06<06 u|U\5<E F0606060606060606rrceZdZdZ ddedejjdeejdeejd eejd e d e ffd Z ej d ej ej dej dfdejdejdejdejdejdejdeejejejejffdZxZS)ConformerEncoderLayeraEncoder layer module. Args: size (int): Input dimension. self_attn (torch.nn.Module): Self-attention module instance. `MultiHeadedAttention` or `RelPositionMultiHeadedAttention` instance can be used as the argument. feed_forward (torch.nn.Module): Feed-forward module instance. `PositionwiseFeedForward` instance can be used as the argument. feed_forward_macaron (torch.nn.Module): Additional feed-forward module instance. `PositionwiseFeedForward` instance can be used as the argument. conv_module (torch.nn.Module): Convolution module instance. `ConvlutionModule` instance can be used as the argument. dropout_rate (float): Dropout rate. normalize_before (bool): True: use layer_norm before each sub-block. False: use layer_norm after each sub-block. N皙?Trr r feed_forward_macaron conv_moduler r c t||_||_||_||_t j|d|_t j|d|_ |#t j|d|_ d|_ nd|_ |j6t j|d|_ t j|d|_ t j||_||_||_dS)rrrNg?g?)rrr r rBrCrrnorm_ffnorm_mhanorm_ff_macaronff_scale norm_conv norm_finalrrrr ) rrr r rBrCr r rs rrzConformerEncoderLayer.__init__s "($8!&|De444  Tu555  +#%<%#@#@#@D DMMDM   '\$E:::DN l%!!!DOz,//  0rrrr r!r"r#r$r%r&r'c|jm|}|jr||}||j|||zz}|js||}|}|jr||}|||||||\}} |||z}|js||}tjd|j |j } |j l|}|jr| |}| |||\}} |||z}|js| |}|}|jr| |}||j|||zz}|js| |}|j ||}||| | fS)u Compute encoded features. Args: x (torch.Tensor): (#batch, time, size) mask (torch.Tensor): Mask tensor for the input (#batch, time，time), (0, 0, 0) means fake mask. pos_emb (torch.Tensor): positional encoding, must not be None for ConformerEncoderLayer. mask_pad (torch.Tensor): batch padding mask used for conv module. (#batch, 1，time), (0, 0, 0) means fake mask. att_cache (torch.Tensor): Cache tensor of the KEY & VALUE (#batch=1, head, cache_t1, d_k * 2), head * d_k == size. cnn_cache (torch.Tensor): Convolution cache in conformer layer (#batch=1, size, cache_t2) Returns: torch.Tensor: Output tensor (#batch, time, size). torch.Tensor: Mask tensor (#batch, time, time). torch.Tensor: att_cache tensor, (#batch=1, head, cache_t1 + time, d_k * 2). torch.Tensor: cnn_cahce tensor (#batch, size, cache_t2). Nrr*)rBr rGrHrrFr r,r-rr+rCrIrEr rJ) rr!r"r#r$r%r&r.r/r0 new_cnn_caches rr2zConformerEncoderLayer.forwards@  $ 0H$ ,((++4=4<<))!,,,.,...A( ,((++   ! a  A#~~aAtW.7 9 9} t||E** *$ ! a  A IQWQXNNN   'H$ &NN1%%#//8YGG A}4<<??*A( &NN1%%    QA t}t||D4E4Ea4H4H'I'II I$  QA   '""A$ }44r)NNNrAT)r3r4r5r6r7r,rr8rr9r:rr;r-r<rr2r=r>s@rr@r@ms.-148+/!!%1118?1ry) 1 'ry1 1 bi( 11111111H",IUZ!H!H!H"-%+l";";"-%+l";";L5L5 <L5lL5 L5 , L5 < L5<L5 u|U\5<E FL5L5L5L5L5L5L5L5rr@) r6typingrrr,rr8rr@rrrOs /."""""""" R6R6R6R6R6biR6R6R6j55555BI55555r