0;ji~5dZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z d d lmZmZmZmZd d lmZd d lmZGddejZGddejZGddejjZdS)zEncoder definition.)TupleN)nn) functional)ConvolutionModule)ConformerEncoderLayer)PositionwiseFeedForward)COSYVOICE_EMB_CLASSESCOSYVOICE_SUBSAMPLE_CLASSESCOSYVOICE_ATTENTION_CLASSESCOSYVOICE_ACTIVATION_CLASSES) make_pad_mask)add_optional_chunk_maskcVeZdZdZd dededeffd Zdejdejfd ZxZ S) Upsample1DaA 1D upsampling layer with an optional convolution. Parameters: channels (`int`): number of channels in the inputs and outputs. use_conv (`bool`, default `False`): option to use a convolution. use_conv_transpose (`bool`, default `False`): option to use a convolution transpose. out_channels (`int`, optional): number of output channels. Defaults to `channels`. r channels out_channelsstridect||_||_||_t j|j|j|dzdzdd|_dS)Nr rr)rpadding)super__init__rrrrConv1dconv)selfrrr __class__s n/root/voice-cloning/.venv/lib/python3.11/site-packages/chatterbox/models/s3gen/transformer/upsample_encoder.pyrzUpsample1D.__init__3s^   ( IdmT-> QWXbcddd inputs input_lengthsctj|t|jd}tj||jdzdfd}||}|||jzfS)Nnearest) scale_factormoder r)value)F interpolatefloatrpadr)rr r!outputss rforwardzUpsample1D.forward;sf-U4;5G5GiXXX%$+/1!5SAAA))G$$  333r)r ) __name__ __module__ __qualname____doc__intrtorchTensorr- __classcell__rs@rrr%s  eeeCeeeeeee4el45<44444444rrcNeZdZddedeffd ZdejdejfdZxZS) PreLookaheadLayerrrpre_lookahead_lenct||_||_t j|||dzdd|_t j||ddd|_dS)Nrr) kernel_sizerr)rrrr9rrconv1conv2)rrr9rs rrzPreLookaheadLayer.__init__Cs|   !2Y h)A-a    Y h!Q    rr returnc|dd}tj|d|jfdd}tj||}tj|ddd}||}|dd}||z}|S)z9 inputs: (batch_size, seq_len, channels) rr rconstantr&)r%r')r r) transpose contiguousr(r+r9 leaky_relur=r>)rr r,s rr-zPreLookaheadLayer.forwardQs""1a((3355%!T%;!<:UXYYY,tzz'2233%jDDD**W%%##Aq))4466F"r)r) r.r/r0r2rr3r4r-r5r6s@rr8r8Bsv            elu|rr8c3eZdZ d8dedededededededededededededejj ded ed!ed"ed#ed$ed%ed&ed'ed(ed)ef2fd* Z d+efd,Z d9d.ej d/ej d0ed1ed+e ej ej ff d2Zd.ej d3ej d4ej d5ej d+ej f d6Zd.ej d3ej d4ej d5ej d+ej f d7ZxZS):UpsampleConformerEncoder皙?linearrel_pos_espnetTrFNr rel_selfattnswish batch_norm input_size output_sizeattention_heads linear_units num_blocks dropout_ratepositional_dropout_rateattention_dropout_rate input_layerpos_enc_layer_typenormalize_beforestatic_chunk_sizeuse_dynamic_chunk global_cmvnuse_dynamic_left_chunkpositionwise_conv_kernel_size macaron_styleselfattention_layer_typeactivation_typeuse_cnn_modulecnn_module_kernelcausalcnn_module_normkey_biasgradient_checkpointingc  t|_||_t | |t | ||_ |_tj d|_ | |_ | |_ ||_||_t!|}|||f||f||||ft#dd|_tj  f dt)|D|_t-ddd|_t | |t | ||_tj  f d t)d D|_d S) a Args: input_size (int): input dim output_size (int): dimension of attention attention_heads (int): the number of heads of multi head attention linear_units (int): the hidden units number of position-wise feed forward num_blocks (int): the number of decoder blocks dropout_rate (float): dropout rate attention_dropout_rate (float): dropout rate in attention positional_dropout_rate (float): dropout rate after adding positional encoding input_layer (str): input layer type. optional [linear, conv2d, conv2d6, conv2d8] pos_enc_layer_type (str): Encoder positional encoding layer type. opitonal [abs_pos, scaled_abs_pos, rel_pos, no_pos] normalize_before (bool): True: use layer_norm before each sub-block of a layer. False: use layer_norm after each sub-block of a layer. static_chunk_size (int): chunk size for static chunk training and decoding use_dynamic_chunk (bool): whether use dynamic chunk size for training or not, You can only use fixed chunk(chunk_size > 0) or dyanmic chunk size(use_dynamic_chunk = True) global_cmvn (Optional[torch.nn.Module]): Optional GlobalCMVN module use_dynamic_left_chunk (bool): whether use dynamic left chunk in dynamic chunk training key_bias: whether use bias in attention.linear_k, False for whisper models. gradient_checkpointing: rerunning a forward-pass segment for each checkpointed segment during backward. gh㈵>)epsrGr<)rr9c  g|]A}tt tr tnd r tndBSNrr r r .0_convolution_layer_argsrWencoder_selfattn_layer_argsrbr\rSpositionwise_layer_argsrcres r z5UpsampleConformerEncoder.__init__..s - - -  "+,DE02')@A1>I',..DH0>I!+--DH    - - - rr )rrrc  g|]A}tt tr tnd r tndBSrnrorps rrvz5UpsampleConformerEncoder.__init__..s 0 0 0  "+,DE02')@A1>I',..DH0>I!+--DH    0 0 0 rN)rr _output_sizer_r r embedr\r3r LayerNorm after_normr]r^r`rjrr8pre_lookahead_layer ModuleListrangeencodersrup_layerup_embed up_encoders)rrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirj activationrsrtrurs ` ` ` `` ` @@@rrz!UpsampleConformerEncoder.__init__esWv '&0=    !"4 5k6M O O    !1(,,[d,CC!2!2&<#&<#1/BDD    "  ' #     # #./@*"16";#4cUV#W#W#W ++ - - - - - - - - - - - - Z(( - - -    #Cc!LLL 3K@    !"4 5k6M O O    !8.. 0 0 0 0 0 0 0 0 0 0 0 0 Qxx 0 0 0   rr?c|jSrn)ry)rs rrSz$UpsampleConformerEncoder.output_sizes   rxsxs_lensdecoding_chunk_sizenum_decoding_left_chunksc |d}t||d}|j||}|||\}}}|}t |||j|j||j|} | |}| || ||}| dd }| ||\}}| dd }|d}t||d}|||\}}}|}t |||j|j||j|j jz|} ||| ||}|jr||}||fS)aEmbed positions in tensor. Args: xs: padded input tensor (B, T, D) xs_lens: input length (B) decoding_chunk_size: decoding chunk size for dynamic chunk 0: default for training, use random dynamic chunk. <0: for decoding, use full chunk. >0: for decoding, use fixed chunk size as set. num_decoding_left_chunks: number of left chunks, this is for decoding, the chunk size is decoding_chunk_size. >=0: use num_decoding_left_chunks <0: use all left chunks Returns: encoder output tensor xs, and subsampled masks xs: padded output tensor (B, T' ~= T/subsample_rate, D) masks: torch.Tensor batch padding mask after subsample (B, 1, T' ~= T/subsample_rate) NOTE(xcsong): We pass the `__call__` method of the modules instead of `forward` to the checkpointing API because `__call__` attaches all the hooks of the module. https://discuss.pytorch.org/t/any-different-between-model-input-and-model-forward-input/3690/2 rNr )sizer unsqueezer_rzrr^r`r]r}forward_layersrBrCrrrforward_up_layersr\r|) rrrrrTmaskspos_embmask_pad chunk_maskss rr-z UpsampleConformerEncoder.forwards< GGAJJw**44Q777   '!!"%%B!ZZE22GU-b%.2.D.2.I.A.2.D.F HH  % %b ) )  ['8 D D\\!Q   * * , ,mmB00 G \\!Q   * * , , GGAJJw**44Q777!]]2u55GU-b%.2.D.2.I.A.2.Dt}G[.[.F HH  # #B Wh G G   %$$B5yrrrrc@|jD]}|||||\}}}}|Srn)rrrrrrlayerrrs rrz'UpsampleConformerEncoder.forward_layers2s<] N NE$)E"k7H$M$M !B Q rc@|jD]}|||||\}}}}|Srn)rrs rrz*UpsampleConformerEncoder.forward_up_layers9s=% N NE$)E"k7H$M$M !B Q r)rGrGrHrIrJrKrKrKrLrMTrFNFrFrNrOFrPFrQTF)rr)r.r/r0r2r*strboolr3rModulerrSr4rr-rrr5r6s@rrFrFcs  !),(+#"2!%!""''+',-.#(6&$!#+',5C C C C  C  C  C C "'C !&C C  C C C  C X_C !%!C "(+#C $%C &#&'C ()C *+C ,-C ./C 01C 23C 4!%5C C C C C C J!S!!!!$%(* CC LCC! C #& C u|U\) * CCCCJEL % !&27,ELu|#(<$)L5:\rrF)r1typingrr3rtorch.nnrr( convolutionr encoder_layerrpositionwise_feed_forwardr utils.class_utilsr r r r utils.maskrrrrr8rFrrrs" $$$$$$******000000>>>>>> '&&&&&00000044444444: B[[[[[ux[[[[[r