Transformers documentation
WavTokenizer
This model was published in HF papers on 2024-08-29 and contributed to Hugging Face Transformers on 2026-11-16.
WavTokenizer
Overview
WavTokenizer encodes 24 kHz mono audio into discrete tokens using a single codebook, at 40 or 75 tokens per second depending on the checkpoint. It can also reconstruct audio from those tokens.
This implementation supports inference with WavTokenizerModel for encoding and decoding, or WavTokenizerEncoderModel for encoding only. The 40-token-per-second variant supplies the audio tokenizer for Apertus 1.5. Training the codec is not supported.
This model was contributed by the Swiss AI Initiative. The original implementation is available on GitHub.
Available checkpoints
These checkpoints are ready to load with from_pretrained:
| Checkpoint | Domain | Tokens per second |
|---|---|---|
| Small, 40 tokens/s | Speech | 40 |
| Small, 75 tokens/s | Speech | 75 |
| Medium | Speech | 75 |
| Medium v2 | Speech | 75 |
| Medium | Music/audio | 75 |
| Medium v2 | Music/audio | 75 |
| Large unified | Unified | 40 |
| Large v2 | Speech | 75 |
Usage example
Load a speech clip, resample it to the checkpoint’s sampling rate, and encode and reconstruct it:
import torch
from transformers import AutoFeatureExtractor, WavTokenizerModel
from transformers.audio_utils import load_audio
model_id = "swiss-ai/wavtokenizer-large-unify-40token"
model = WavTokenizerModel.from_pretrained(model_id)
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
audio = load_audio(
"https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/belinda.wav",
sampling_rate=feature_extractor.sampling_rate,
)
inputs = feature_extractor(audio=audio, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt")
with torch.no_grad():
audio_codes = model.encode(**inputs).audio_codes
reconstruction = model.decode(audio_codes).audio_values[..., : len(audio)]Usage notes
- Audio inputs: The feature extractor expects mono audio at its configured sampling rate; it does not
resample or downmix. The example uses
load_audioto handle both. Decoding can add trailing samples due to internal padding, so the example trims the reconstruction to the original length. - Batching: Padding clips of different lengths can change their codes. Encode clips separately when you need the same codes as processing each clip on its own.
- Precision: Load and run the tokenizer in
float32(the default). Lower precision can change code assignments.
WavTokenizerConfig
class transformers.WavTokenizerConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonesampling_rate: int = 24000audio_channels: int = 1num_filters: int = 32upsampling_ratios: list[int] | tuple[int, ...] = (6, 5, 5, 4)num_residual_layers: int = 1dilation_growth_rate: int = 2compress: int = 2use_conv_shortcut: bool = Trueuse_causal_conv: bool = Falsepad_mode: str = 'reflect'norm_type: str = 'weight_norm'kernel_size: int = 7last_kernel_size: int = 7residual_kernel_size: int = 3num_lstm_layers: int = 2hidden_size: int = 512codebook_size: int = 4096codebook_dim: int = 512decoder_hidden_size: int = 768decoder_intermediate_size: int = 2304decoder_num_layers: int = 12adanorm_num_embeddings: int = 4decoder_attention_num_groups: int = 32norm_eps: float = 1e-06 )
Parameters
- sampling_rate (
int, optional, defaults to24000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). - audio_channels (
int, optional, defaults to 1) — Number of channels in the audio data. Only mono (1) is supported. - num_filters (
int, optional, defaults to 32) — Number of convolution kernels of the firstWavTokenizerConv1ddown sampling layer. - upsampling_ratios (
list[int], optional, defaults to[6, 5, 5, 4]) — Kernel size and stride ratios. The encoder uses downsampling ratios instead of upsampling ratios, hence it will use the ratios in the reverse order to the ones specified here that must match the decoder order. The product (600 by default) is the hop length: the number of audio samples represented by a single code. - num_residual_layers (
int, optional, defaults to 1) — Number of residual layers in each encoder stage. - dilation_growth_rate (
int, optional, defaults to 2) — How much to increase the dilation with each residual layer in the encoder. - compress (
int, optional, defaults to 2) — Reduced dimensionality in residual branches. - use_conv_shortcut (
bool, optional, defaults toTrue) — Whether to use a convolutional layer as the (skip) shortcut in the residual blocks. If False, an identity function will be used, giving a generic residual connection. - use_causal_conv (
bool, optional, defaults toFalse) — Whether to use fully causal convolution. - pad_mode (
str, optional, defaults to"reflect") — Padding mode for the convolutions. - norm_type (
str, optional, defaults to"weight_norm") — Normalization method for the encoder convolutions. Should be in["weight_norm", "time_group_norm"]. - kernel_size (
int, optional, defaults to 7) — Kernel size for the initial encoder convolution. - last_kernel_size (
int, optional, defaults to 7) — Kernel size for the last encoder convolution. - residual_kernel_size (
int, optional, defaults to 3) — Kernel size for the residual layers of the encoder. - num_lstm_layers (
int, optional, defaults to 2) — Number of LSTM layers at the end of the encoder. - hidden_size (
int, optional, defaults to 512) — Dimensionality of the encoder output (the latent space of the quantizer). - codebook_size (
int, optional, defaults to 4096) — Number of discrete codes that make up the single VQ codebook. - codebook_dim (
int, optional, defaults to 512) — Dimension of the codebook vectors. Must matchhidden_size(no projections are used). - decoder_hidden_size (
int, optional, defaults to 768) — Dimensionality of the Vocos-style decoder backbone. - decoder_intermediate_size (
int, optional, defaults to 2304) — Dimensionality of the pointwise convolutions inside the decoder ConvNeXt blocks. - decoder_num_layers (
int, optional, defaults to 12) — Number of ConvNeXt blocks in the decoder backbone. - adanorm_num_embeddings (
int, optional, defaults to 4) — Number of condition embeddings of the decoder’s adaptive layer norms. Conditioning is a training-time feature (one embedding per bandwidth); inference always uses embedding 0. - decoder_attention_num_groups (
int, optional, defaults to 32) — Number of groups of the decoder GroupNorm layers. - norm_eps (
float, optional, defaults to 1e-06) — Epsilon of the decoder normalization layers.
This is the configuration class to store the configuration of a WavTokenizerModel. It is used to instantiate a Wavtokenizer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the swiss-ai/wavtokenizer-large-unify-40token
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
Example:
>>> from transformers import WavTokenizerConfig, WavTokenizerModel
>>> # Initializing configuration
>>> configuration = WavTokenizerConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = WavTokenizerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configWavTokenizerEncoderConfig
class transformers.WavTokenizerEncoderConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonesampling_rate: int = 24000audio_channels: int = 1num_filters: int = 32upsampling_ratios: list[int] | tuple[int, ...] = (6, 5, 5, 4)num_residual_layers: int = 1dilation_growth_rate: int = 2compress: int = 2use_conv_shortcut: bool = Trueuse_causal_conv: bool = Falsepad_mode: str = 'reflect'norm_type: str = 'weight_norm'kernel_size: int = 7last_kernel_size: int = 7residual_kernel_size: int = 3num_lstm_layers: int = 2hidden_size: int = 512codebook_size: int = 4096codebook_dim: int = 512decoder_hidden_size: int = 768decoder_intermediate_size: int = 2304decoder_num_layers: int = 12adanorm_num_embeddings: int = 4decoder_attention_num_groups: int = 32norm_eps: float = 1e-06 )
Configuration of WavTokenizerEncoderModel, the encoder and quantizer of WavTokenizer. It shares every field with WavTokenizerConfig (see its documentation for the arguments), so it can be loaded from a full WavTokenizer checkpoint. Decoder-specific fields are unused by the encoder model.
WavTokenizerFeatureExtractor
class transformers.WavTokenizerFeatureExtractor
< source >( feature_size: int = 1sampling_rate: int = 24000padding_value: float = 0.0hop_length: int = 600normalize_audio: bool = Falsetarget_peak_dB_FS: float = -3.0**kwargs )
Parameters
- feature_size (
int, optional, defaults to 1) — The feature dimension of the extracted features. Only mono (1) is supported. - sampling_rate (
int, optional, defaults to 24000) — The sampling rate at which the audio waveform should be digitalized, expressed in hertz (Hz). - padding_value (
float, optional, defaults to 0.0) — The value that is used for padding. - hop_length (
int, optional, defaults to 600) — Number of audio samples represented by a single audio code. - normalize_audio (
bool, optional, defaults toFalse) — Whether to peak-normalize each clip totarget_peak_dB_FSbefore padding. The original WavTokenizer does not normalize; Apertus 1.5 enables it. - target_peak_dB_FS (
float, optional, defaults to -3.0) — Target peak level in dB FS used whennormalize_audioisTrue.
Constructs a WavTokenizer feature extractor for mono audio at the configured sampling rate. Audio is not resampled or downmixed.
__call__
< source >( audio: typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor']]padding: bool | str | transformers.utils.generic.PaddingStrategy | None = Nonetruncation: bool | None = Falsemax_length: int | None = Nonereturn_tensors: str | transformers.utils.generic.TensorType | None = Nonesampling_rate: int | None = None ) → BatchFeature
Parameters
- audio (
np.ndarray,torch.Tensor,list[float],list[np.ndarray],list[torch.Tensor],list[list[float]]) — The sequence or batch of sequences to be processed. Each sequence must be mono audio of shape(num_samples,), already sampled at this feature extractor’ssampling_rate. Audio is not resampled or downmixed. - padding (
bool,stror PaddingStrategy, optional, defaults toTrue) — Padding strategy:Trueor'longest'pads a batch to its longest sequence (single inputs stay unpadded),'max_length'pads tomax_length,Falseor'do_not_pad'disables padding. - truncation (
bool, optional, defaults toFalse) — Whether to truncate sequences longer thanmax_lengthtomax_length. - max_length (
int, optional) — Target length for'max_length'padding and fortruncation. - return_tensors (
stror TensorType, optional) — If set, return tensors instead of a list of numpy arrays:'pt'for PyTorchtorch.Tensor,'np'for NumPynp.ndarray. - sampling_rate (
int, optional) — Sampling rate of theaudioinput; pass it so a mismatch with the configuredsampling_rateraises an error instead of failing silently.
Returns
A BatchFeature with the following fields:
- input_values — Audio waveforms to be fed to the model, as a list of arrays of shape
(1, num_samples), or a tensor of shape(batch_size, 1, num_samples)whenreturn_tensorsis set. - padding_mask — Mask indicating valid audio samples (1) versus padding (0), of shape
(batch_size, num_samples).
Featurize and prepare one or several audio sequence(s) for the model.
WavTokenizerEncoderModel
class transformers.WavTokenizerEncoderModel
< source >( config: WavTokenizerEncoderConfig )
Parameters
- config (WavTokenizerEncoderConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
Encoder and quantizer of WavTokenizer for converting audio waveforms to discrete codes.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
encode
< source >( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = None**kwargs ) → WavTokenizerEncoderOutput or tuple(torch.Tensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, 1, sequence_length)) — Input audio waveform. Arbitrary non-zero lengths are supported; the encoder pads internally and emitsceil(sequence_length / hop_length)codes. - padding_mask (
torch.Tensorof shape(batch_size, sequence_length)or(batch_size, 1, sequence_length), optional) — Mask describing padding ininput_values; used to computeaudio_codes_mask.
Returns
WavTokenizerEncoderOutput or tuple(torch.Tensor)
A WavTokenizerEncoderOutput or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (None) and inputs.
- audio_codes (
torch.LongTensorof shape(batch_size, 1, codes_length), optional) — Discrete code indices computed usingmodel.encode. These represent the compressed, quantized form of the input audio signal (one codebook,ceil(num_samples / hop_length)codes). - audio_codes_mask (
torch.Tensorof shape(batch_size, 1, codes_length), optional) — Downsampledpadding_maskindicating valid audio codes inaudio_codes.
forward
< source >( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = None**kwargs ) → WavTokenizerEncoderOutput or tuple(torch.Tensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, 1, sequence_length)) — Input audio waveform. - padding_mask (
torch.Tensorof shape(batch_size, sequence_length)or(batch_size, 1, sequence_length), optional) — Padding mask used to computeaudio_codes_mask.
Returns
WavTokenizerEncoderOutput or tuple(torch.Tensor)
A WavTokenizerEncoderOutput or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (WavTokenizerConfig) and inputs.
The WavTokenizerEncoderModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
- audio_codes (
torch.LongTensorof shape(batch_size, 1, codes_length), optional) — Discrete code indices computed usingmodel.encode. These represent the compressed, quantized form of the input audio signal (one codebook,ceil(num_samples / hop_length)codes). - audio_codes_mask (
torch.Tensorof shape(batch_size, 1, codes_length), optional) — Downsampledpadding_maskindicating valid audio codes inaudio_codes.
WavTokenizerModel
class transformers.WavTokenizerModel
< source >( config: WavTokenizerConfig )
Parameters
- config (WavTokenizerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
Inference-only, single-codebook WavTokenizer neural audio codec.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
decode
< source >( audio_codes: Tensorbandwidth_id: int = 0**kwargs ) → WavTokenizerDecoderOutput or tuple(torch.Tensor)
Parameters
- audio_codes (
torch.LongTensorof shape(batch_size, 1, codes_length)) — Discrete code indices computed usingmodel.encode. One or more codes are supported; each code decodes tohop_lengthaudio samples. - bandwidth_id (
int, optional, defaults to 0) — Condition embedding id of the decoder’s adaptive layer norms. Always 0 at inference.
Returns
WavTokenizerDecoderOutput or tuple(torch.Tensor)
A WavTokenizerDecoderOutput or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (None) and inputs.
- audio_values (
torch.Tensorof shape(batch_size, 1, sequence_length), optional) — Decoded audio waveform values in the time domain (codes_length * hop_lengthsamples).
encode
< source >( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = None**kwargs ) → WavTokenizerEncoderOutput or tuple(torch.Tensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, 1, sequence_length)) — Input audio waveform. Arbitrary non-zero lengths are supported; the encoder pads internally and emitsceil(sequence_length / hop_length)codes. - padding_mask (
torch.Tensorof shape(batch_size, sequence_length)or(batch_size, 1, sequence_length), optional) — Mask describing padding ininput_values; used to computeaudio_codes_mask.
Returns
WavTokenizerEncoderOutput or tuple(torch.Tensor)
A WavTokenizerEncoderOutput or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (None) and inputs.
- audio_codes (
torch.LongTensorof shape(batch_size, 1, codes_length), optional) — Discrete code indices computed usingmodel.encode. These represent the compressed, quantized form of the input audio signal (one codebook,ceil(num_samples / hop_length)codes). - audio_codes_mask (
torch.Tensorof shape(batch_size, 1, codes_length), optional) — Downsampledpadding_maskindicating valid audio codes inaudio_codes.
forward
< source >( input_values: Tensorpadding_mask: typing.Optional[torch.Tensor] = None**kwargs ) → WavTokenizerOutput or tuple(torch.Tensor)
Parameters
- input_values (
torch.Tensorof shape(batch_size, 1, sequence_length)) — Input audio waveform. - padding_mask (
torch.Tensorof shape(batch_size, sequence_length)or(batch_size, 1, sequence_length), optional) — Mask indicating valid audio samples (1) versus padding (0), used to computeaudio_codes_mask.
Returns
WavTokenizerOutput or tuple(torch.Tensor)
A WavTokenizerOutput or a tuple of
torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (None) and inputs.
The WavTokenizerModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
- audio_values (
torch.Tensorof shape(batch_size, 1, sequence_length), optional) — Decoded audio waveform values in the time domain, obtained using the decoder part of WavTokenizer. - audio_codes (
torch.LongTensorof shape(batch_size, 1, codes_length), optional) — Discrete code indices computed usingmodel.encode. - audio_codes_mask (
torch.Tensorof shape(batch_size, 1, codes_length), optional) — Downsampledpadding_maskindicating valid audio codes inaudio_codes.