Peer-reviewed research
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Changli Tang et al., arXiv (ICLR 2024), 2023
Read the originalRead on October 17, 2026
- What it is
- A model from Tsinghua University and ByteDance that joins speech and audio encoders to a large language model, so the language model can process general audio and perform tasks such as automatic speech recognition.
- How much weight it carries
- Accepted at ICLR 2024 and widely cited. The authors report results on their own benchmarks and training tasks, so it shows that the approach works, not how it compares with the best dedicated recognizers.
Facts taken from this source
- The SALMONN authors say their model lets a large language model process and understand general audio inputs directly, with competitive performance on speech tasks such as automatic speech recognition.
“SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as automatic speech recognition and translation” (Abstract)