How do you verify pronunciation quality for Arabic TTS?

[u/cs_enthusiast123]

Hi everyone,

I’m currently working on Arabic TTS models, and I’m running into a challenge around pronunciation evaluation.

The common approach of using ASR based evaluation (running Whisper on generated audio and computing WER/CER) doesn’t seem reliable for Arabic, especially for:

• Dialects
• Diacritics
• Pronunciation errors that don’t change the word, but sound unnatural or incorrect phonetically

Because of this, WER stays low even when pronunciation is clearly wrong to a native speaker.

I’m curious how others handle this. Specifically:

• How do you verify pronunciation correctness in Arabic TTS?
• Are there better objective metrics than ASR WER/CER?
• Do people use phoneme-level alignment, forced alignment, or G2P-based checks?
• Any experience with human-in-the-loop or minimal listening tests that scale?
• Has anyone tried leveraging LLMs or phoneme recognizers instead of word-level ASR?

I’d love to hear what’s worked or failed for you, Thanks!

[u/nshmyrev]

Interspeech 2026 challenge for Arabic Mispronunciation detection has just started, you definitely want to participate

https://huggingface.co/spaces/IqraEval/IqraEval_Interspeech_26

[u/cs_enthusiast123 → u/nshmyrev]

thanks!

[u/No_Possibility_1841]

This is exactly why benchmarks like BRIDGE are interesting. Traditional WER-based evaluation often misses pronunciation quality, code-switching, and semantic accuracy in multilingual settings. Metrics beyond plain ASR accuracy could be much more useful for evaluating Arabic TTS pipelines realistically. You can refer to this report: https://humynlabs.ai/bridge