0;ji+MdZddlmZddlmZmZmZddlZddl Z ddl m cm Z ddlmZddl mZm Z ddlmZmZmZmZeGd d ZGd d e jZGd de jZGdde jZddZGdde jZGdde jZGdde jZGdde jZGdde jZ Gdde jZ!dS) zyModified from https://github.com/openai/whisper/blob/main/whisper/model.py Add EuclideanCodebook & VectorQuantization ) dataclass)IterableOptionalTupleN rearrange)Tensornn)make_non_pad_mask mask_to_bias onnx2torchmerge_tokenized_segmentscreZdZUdZeed<dZeed<dZeed<dZeed<d Z eed <d Z eed <d Z e ed<dS) ModelConfign_melsi n_audio_ctxi n_audio_state n_audio_head n_audio_layerin_codebook_sizeFuse_sdpaN) __name__ __module__ __qualname__rint__annotations__rrrrrrboolK/root/voice-cloning/.venv/lib/python3.11/site-packages/s3tokenizer/model.pyrrsFCKM3L#M3OSHdr#rceZdZdedefdZdS) LayerNormxreturnctj||j|j|jnd|j|jnd|j|jSN) F layer_normfloatnormalized_shapeweightbiasepstypedtypeselfr's r$forwardzLayerNorm.forward-ss| GGII  !#';#:DK     !%!6DIOO   D H   $qw--  r#Nrrrr r6r"r#r$r&r&+s6Fr#r&ceZdZdedefdZdS)Linearr'r(ctj||j|j|jdn|j|jSr*)r+linearr/tor3r0r4s r$r6zLinear.forward9sJx KNN17 # #I%DD49<<+@+@   r#Nr7r"r#r$r9r97s6  F      r#r9c<eZdZdededeedeffd ZxZS)Conv1dr'r/r0r(ct|||j|dn||jSr*)super _conv_forwardr<r3)r5r'r/r0 __class__s r$rAzConv1d._conv_forwardCsMww$$ vyy!!4<44TWWQW=M=MOO Or#)rrrr rrA __classcell__rBs@r$r>r>AsnOvOvO$V,O17OOOOOOOOOOr#r>'c|dzdksJtj||dzdz z }tj| tj|dzz}tj|ddtjf|tjddfz}tjtj|tj|gdS)z*Returns sinusoids for positional embeddingrr Ndim) nplogtorchexparangenewaxiscatsincos)lengthchannels max_timescalelog_timescale_incrementinv_timescales scaled_times r$ sinusoidsrYIs a<1     f]33x1}q7HIY 77$|HM:: ;<z)AudioEncoder.__init__..s>G G G  #7FX F F FG G G r#) r@r_rr>conv1conv2register_bufferrYr ModuleListrangeblocks) r5rrr\r]rrrrBs `` `r$r_zAudioEncoder.__init__s  F#()#)$% ''' G#()#$$% ''' 3Yug5N5NOOO8: G G G G G G 7^^G G G 9 9  r#r'x_lenr(ct|d}tj|||z}|dzdz dz |jzdz}t|d}tj|||z}|dzdz dz dzdz}t|d}|ddd}t||j }||j d|j dddfz |j }|j D]!}|||d}"||fS)z x : torch.Tensor, shape = (batch_size, n_mels, T) the mel spectrogram of the audio x_len: torch.Tensor, shape = (batch_size,) length of each audio in x r rGrN)r unsqueezer+gelurrrrur r3rrsr<r)r5r'rrdblocks r$r6zAudioEncoder.forwardsb!''11!44 F4::a$h'' ( ([(1,: : > >qw G G[ , ,Ea**++AA%xr#) rrrrr!r_r rr6rCrDs@r$rrs                :53Hr#rc~eZdZdZdedeffd ZejdedefdZ ejdedefdZ ejd Z ejd edefd Z ejdedefd Z ejd edefd ZxZS)EuclideanCodebookzCodebook with Euclidean distance (inference-only). Args: dim (int): Dimension. codebook_size (int): Codebook size. rI codebook_sizecttj||}||_|d|dS)Nembed)r@r_rLzerosrr)r5rIrrrBs r$r_zEuclideanCodebook.__init__sN  M3//* We,,,,,r#r'r(c&t|d}|S)Nz... d -> (...) drr4s r$ preprocesszEuclideanCodebook.preprocesss a+ , ,r#c\|j|j}|dddd|z|zz |dddz }|dj}|S)NrGr T)keepdimrrmrH)rtr<r3powsummaxindices)r5r'rdist embed_inds r$quantizezEuclideanCodebook.quantizes !!!'**qa..Q>1!!!T!2234HHH$$, r#c&|j|ddS)Nrm)rt)r5rrss r$postprocess_embz!EuclideanCodebook.postprocess_embsy~uSbSz**r#rc:tj||j}|Sr*)r+ embeddingrr5rrs r$ dequantizezEuclideanCodebook.dequantizes;y$*55r#c|j}||}||}|||}|Sr*)rsrrr)r5r'rsrs r$encodezEuclideanCodebook.encodesF OOA  MM!$$ ((E:: r#c0||}|Sr*)rrs r$decodezEuclideanCodebook.decodes??9--r#)rrr__doc__rr_rLinference_moder rrrrrrrCrDs@r$rrs -C------- UFvU&VU+++UFvU6U6r#rceZdZdZdedeffd ZedZej de de fdZ ej d e de fd Z xZ S) VectorQuantizationzVector quantization implementation (inference-only). Args: dim (int): Dimension codebook_size (int): Codebook size rIrctt|||_||_dS)N)rIr)r@r_r _codebookr)r5rIrrBs r$r_zVectorQuantization.__init__sE *s9FHHH*r#c|jjSr*)rrr5s r$codebookzVectorQuantization.codebook"s ~##r#r'r(ctj|dd}|j|}|S)NrGrm)prI)r+ normalizer-rr)r5r'embed_ins r$rzVectorQuantization.encode&s: K QB / / />((++r#rcZ|j|}t|d}|S)Nzb n d -> b d n)rrrrs r$rzVectorQuantization.decode,s,>((33X'788r#)rrrrrr_propertyrrLrr rrrCrDs@r$rrs +C+++++++ $$X$U6 U6r#rc <eZdZdZefdedeffd ZdededeeeffdZ e j dededeeeffd Z e j deded ed e deeeff d Zed ZdefdZdefdZdZxZS) S3TokenizerzbS3 tokenizer implementation (inference-only). Args: config (ModelConfig): Config nameconfigc rt||_||_t |jj|jj|jj|jj|jj |dkrdnd|jj |_ t|jj|jj |_dS)Nspeech_tokenizer_v1_25hzrGr )r@r_rrrrrrrrrencoderrr quantizer)r5rrrBs r$r_zS3Tokenizer.__init__9s   # K  K # K % K $ K %333AA K    ,DK,E,0K,GIIr#melmel_lenr(c.|||Sr*)r)r5rrs r$r6zS3Tokenizer.forwardIs}}S'***r#cd}||k}|r|||||S|||\}}|j|}||fS)ap Quantize mel spectrogram to tokens, with automatic long audio handling. Args: mel: mel spectrogram tensor, shape (batch_size, n_mels, T) mel_len: mel length tensor, shape (batch_size,) Returns: code: quantized tokens, shape (batch_size, T') code_len: token length, shape (batch_size,) i )any_quantize_mixed_batchrrr)r5rr max_frameslong_audio_maskhiddencode_lencodes r$rzS3Tokenizer.quantizeLs "J.     "--c7O.8:: : $||C99 FH>((00D> !r#rrc |d}d}d}d}d} ||z|z} | |z|z} | | z } g} g}g}t|D]}||}||}||}|s|ddd|f}|}|| kr| |z }tj|d|f}| ||t j||j||ddd d d}d}||krt|| z|}|dd||f}|d }|| kr| |z }tj|d|f}| ||t j||j||d |dd |d z }|| z }||k|}|D]}|d |kr |d r||d<| sMt j |dt j |jt j |t j |jfSt j | }t j |}| ||\}}|j|} i}!t!|D]\}"}|d }|d }|d}| |"d||"f}#|s;t j|#t j |j}$|$t)|#f|!|<||!vrg|!|<|!||#t|D]}||rt|!|}%t+|dr|jdkrd}&nd}&t/|%| |&}'t j|'t j |j}(|(t)|'f|!|<t1d|!D})t j ||)t j |j}*t j |t j |j}+|!D]\}\}$}|$|*|d|f<||+|<|*|+fS)a Handle mixed batch with both short and long audio using unified batch processing. Args: mel: mel spectrogram tensor, shape (batch_size, n_mels, T) mel_len: mel length tensor, shape (batch_size,) long_audio_mask: boolean mask for long audio, shape (batch_size,) max_frames: maximum frames for short audio Returns: code: quantized tokens, shape (batch_size, T') code_len: token length, shape (batch_size,) ri>rN)deviceFr ) batch_idx is_long_audio segment_idxtotal_segmentsTrrr)r3rrrspeech_tokenizer_v12)overlap token_ratec3&K|] }|dV dS)r Nr")r code_infos r$ z4S3Tokenizer._quantize_mixed_batch.. s&JJI9Q<JJJJJJr#)r}ritemr+padappendrLtensorrminrlongstackrrr enumeratecpunumpytolistlenhasattrrrrvaluesitems),r5rrrr batch_size sample_rate hop_length window_sizerframes_per_windowframes_per_overlapframes_per_stride all_segmentsall_segments_len segment_infor audio_mel audio_mel_lenrsegmentseg_lenpad_sizestartrendrinfounified_batch_melunified_batch_lensrrcodesresultsseg_idx segment_code code_tensor audio_codesr merged_codesmerged_codes_tensor max_code_len output_codesoutput_codes_lens, r$rz!S3Tokenizer._quantize_mixed_batchjs"XXa[[    (+5C${2j@-0BB  z**7 @7 @III#I.M+I6;;==M 2 @#AAA~ ~$56',,.....07:HeGa];;G##G,,, ''L<<<>>>##!*%*#$&' %% m++e&77GGC'59 5G%ll1ooG!222#4w#>"#%!X"?"? ''000$++ WSZ@@@BBB ''%.)-'2*. )) 1$K..E+m++0"-(@@DK(I55$:O51?-.@ 6;z %*Z&)j22249;#-).*-*4646466 6"K 55"[)9:: <<(9;MNN%%f--&|44 8 8MGT[)I 1M}-K!2(7+002222447CEE%%''&&(( ! 8#l<1625*>>> '23|3D3D%E ""G++)+GI& ")),7777z** N NIy)..00 N%i0 4!##$'+y4I'I'I!#JJ!#J7 @GCM O O O ',l<9>:=*'F'F'F#':3|;L;L%M "JJ9I9IJJJJJ {:#/).*-*666 !;z-2Z.1j:::3:--// 3 3 .I. X1rYModuler[rrrrrr"r#r$r>s"!!!!!,,,,,,,,,, XXXXXXXXXXXX               RY   OOOOORYOOONNNN: : : : : : : : zRY.6666629666r11111 111h