0;ji3ddlmZmZmZddlZddlmZddlmZm Z m Z m Z m Z ddl mZddlmZddlmZGddejZGd d ejZeGd d ejZdS) )AnyDictOptionalN)GEGLUGELU AdaLayerNormAdaLayerNormZeroApproximateGELU) Attention)LoRACompatibleLinear)maybe_allow_in_graphc*eZdZdZdfd ZdZxZS) SnakeBetaam A modified Snake function which uses separate parameters for the magnitude of the periodic components Shape: - Input: (B, C, T) - Output: (B, C, T), same shape as the input Parameters: - alpha - trainable parameter that controls frequency - beta - trainable parameter that controls magnitude References: - This activation function is a modified version based on this paper by Liu Ziyin, Tilman Hartwig, Masahito Ueda: https://arxiv.org/abs/2006.08195 Examples: >>> a1 = snakebeta(256) >>> x = torch.randn(256) >>> x = a1(x) ?Tctt|tr|n|g|_t |||_||_|jrgtj tj |j|z|_ tj tj |j|z|_ nftj tj|j|z|_ tj tj|j|z|_ ||j _||j _d|_dS)a Initialization. INPUT: - in_features: shape of the input - alpha - trainable parameter that controls frequency - beta - trainable parameter that controls magnitude alpha is initialized to 1 by default, higher values = higher-frequency. beta is initialized to 1 by default, higher values = higher-magnitude. alpha will be trained along with the rest of your model. g& .>N)super__init__ isinstancelist in_featuresr projalpha_logscalenn Parametertorchzerosalphabetaones requires_gradno_div_by_zero)selfr out_featuresralpha_trainabler __class__s d/root/voice-cloning/.venv/lib/python3.11/site-packages/chatterbox/models/s3gen/matcha/transformer.pyrzSnakeBeta.__init__#s +5lD+I+I]<