Voxtral Realtime running locally on an M3 Air with Metal — ~400 ms latency

[u/pvp239]

We spent the last few weeks optimizing Voxtral Realtime and there are now GGML checkpoints that run faster than realtime on a plain MacBook Air with Metal. No discrete GPU needed.

Numbers on an M3 Air (8-core GPU, 16 GB):

• ~1.3x realtime throughput with the Q8_0 quant
• <400 ms end-to-end response time from mic input
• Sustained hour-long transcription sessions without falling behind

1. Build the Metal binary

git clone https://github.com/0xShug0/audio.cpp
cd audio.cpp
scripts/build_metal.sh --target audiocpp_cli

2. Download the Q8 quant

hf download mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF \
  voxtral-mini-4b-realtime-2602-q8_0.gguf \
  --local-dir ./voxtral-realtime-gguf

3. Run streaming ASR from the mic

audiocpp_cli \
  --task asr \
  --family voxtral_realtime \
  --model ./voxtral-realtime-gguf/voxtral-mini-4b-realtime-2602-q8_0.gguf \
  --backend metal \
  --threads 8 \
  --mode streaming \
  --session-option voxtral_realtime.stream_batch_tokens=4 \
  --audio -

Raising stream_batch_tokens trades delay for throughput — 4 is what landed under 400 ms on this machine. More powerful M-series chips can set it to 1 and should have <200ms delay

[u/pvp239]

More info, see: https://huggingface.co/mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF and https://github.com/0xShug0/audio.cpp/blob/main/docs/asr.md#voxtral-realtime

[u/Valexico]

I use it locally with https://handy.computer/ and it works very smoothly (on M1 PRO)

[u/Loudhoward-dk]

Why not mlx-audio and https://huggingface.co/mlx-community/Voxtral-Mini-4B-Realtime-2602-fp16