Voice Models
{% hint style="info" %} WIP still, catching up with current open/commercial options and references to DIY {% endhint %}
Current Cascading Approach: VAD → ASR → LLM → TTS
Has problems:
- Interruptions
- First word Latency
- Cascading of errors from VAD/ASR
- emotion, tone, and other speech features are lost
State of the Art:
Full Duplex Models (end to end continuous speech in/out):
Moshi.chat by Kyutai (pending open source/API release)
LSLM (Language Model Can Listen While Speaking) by ByteDance
Turn Based Models with Speech as Input:
Ultravox 0.4 by Fixie
Qwen2-Audio-Chat by Qwen Team, Alibaba (Whisper encoder to embed melspectrogram, then use it as prefix to LLM)
GPT-4o voice by OpenAI
Yet to read papers folder
People to follow ( X.com list)
Notes
Cascading Approach
minimum latency: 500ms on cloud via
VAD: silero-vad on edge/onnx
ASR: whisper via Groq
LLM: Llama via Groq
TTS: Sonic by Cartesia
Or Locally via modular HF pipeline
Ultravox → LLama3 + Whisper Encoder, plans on full duplex, API in beta
Gazelle -> Mistral 7b instruct + wav2vec2 + DPO, API in waitlist
The speech signals are encoded into discrete tokens, and then discrete speech tokens are expanded into the vocabulary of the LLM\
LSLM architecture
Speech/Text to Speech/Text:
non dialogue tasks: LauraGPT, Viola, VoxtLM\
Speech/Text to Text:
- Encoder + Adapter style models: Following the manner of LLaVA, we leverage the well-trained speech modal encoder and the LLM, which makes LLaSM more resource-friendly. Specifically, we use Whisper as a speech encoder to encode the speech signals into embeddings. Then a modal adaptor learns to align speech embeddings with the input text embeddings of the large language model. The speech embeddings and the text embeddings are concatenated together to form interleaved sequences, then the interleaved sequences are input to the LLM for supervised fine-tuning. The training process is divided into two stages. In the first stage, we use the public ASR datasets for the modality adaptation pre-training. The speech encoder and the LLM are frozen, only the modal adaptor is trained to align the speech and text embeddings. As most of the model parameters remain frozen, only a small part of the parameters from the modal adaptor is trained during this stage, it is not resource-consuming. In the second stage, we use cross-modal instruction data for training to provide the model with the capacity to process cross-modal conversations and handle multi-modal instructions. The speech encoder is frozen while the parameters of the modal adaptor and the language model are updated for cross-modal instruction fine-tuning.
Qwen2-Audio-Chat (latest)
WavLLM by Msft
SenseVoice (90 ms)
Audio Flamingo (dialogue trained, tiny base model though)
GAMA Audio
LTU-AS,
Salmon
LLaSM,
COSMIC
<figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXd8OBLeHcyri5y3G0wTGbwktGMUBh6rlsfz0_pHGOBl7TNUldoGVWXq_6LeCISK0IWJ-qMxeVFhSGjVxrhMeswlKDnf0Q-DWDy_wlJbQE6PfXB9WqqXcbCLMDyFpYVEF_MUbev4FsGN5mWuFEk1U5vr6As?key=g3jZRcP1xgntv30Siq4dbQ" alt=""><figcaption></figcaption></figure>
<figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXc_FBklTtxyvXAIfPesf9ekEbMWhRphHDaN815mCtj2bcUUE9FIK1RaamsL-JK2XRgagGpfArK2GKp0v38yGYeuPMBHLgJGGZSkD6tSIQ5bFqZk1CRP1VBVe-Rf79XaaxhMjzdTXoNDzXeBKdBfhNmvaRM?key=g3jZRcP1xgntv30Siq4dbQ" alt=""><figcaption></figcaption></figure>
<figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXcNVZPhLCM6rBEe6yPWKyUkFPJcTKnpaihqepncP0e-6dcphKlgm4RwtRAm0kdX1lCORKJxcep9gGxbnm0XovaSUXOu_OfkKhJ8-qMAVCO_zd1hnZdMlyVPJpcUs3tmN1oni-iy63dvLgxrD2uywrcvH_zr?key=g3jZRcP1xgntv30Siq4dbQ" alt=""><figcaption></figcaption></figure>
VIsion Language Models
- Palm-E: 540B PaLM + 22B Vision Transformer
- LLaVA: pre-trained CLIP visual encoder + LLaMA and instruct tuning on GPT4-assisted visual instruction data.
- BLIP-2: Flan-T5 with a Q-Former to align visual features with the LM\
Cascading Models:
- Parler-tts: high-quality, natural sounding speech with features that can be controlled using a simple text prompt (e.g. gender, background noise, speaking rate, pitch and reverberation) and consistent voices.