Improving the Semantic Structure of Neural Audio Codecs

Abstract

Treball fi de màster de: Master in Sound and Music Computing

Supervisor: Dr. Lonce Wyse

Neural audio codecs have achieved remarkable compression efficiency by learning latent representations optimized for waveform fidelity. However, these codecs often lack explicit semantic structure, limiting their effectiveness for downstream tasks that require meaningful audio abstractions. Query-based compression, as introduced by ALMTokenizer, offers a path to infuse global context into discrete audio tokens by interleaving learnable [CLS] embeddings among frame-level features and leveraging Transformer attention to aggregate semantic information. This thesis implements a reproducible pipeline that adapts the ALMTokenizer paradigm using a frozen EnCodec front-end. By inserting one [CLS] query token every w frames, the model enables bitrate-on-demand through a tunable window length, while a Transformer encoder–decoder architecture captures long-range dependencies and reconstructs waveforms via a paired decoder. Quantization layers are omitted in this implementation to focus analysis on the raw contextual embeddings. To assess the semantic organization of the resulting latent space, we extract [CLS] embeddings from the Good-sounds dataset and perform an evaluation of the resulting latents. Our analyses show that although ALMTokenizer reconstructions lag behind EnCodec in perceptual quality, its embeddings exhibit stronger semantic organization. Clustering, projection, and classification experiments reveal clearer groupings by instrument, note, and octave, while interpolation suggests smoother latent transitions. This highlights a trade-off: EnCodec excels at fidelity, whereas ALMTokenizer provides embeddings better suited for semantic tasks. By releasing the implementation and methodology, this thesis offers a foundation for future research on semantically structured audio codecs.