PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark
Abstract
PARSA-Bench presents the first benchmark for evaluating large audio-language models on Persian language and culture, featuring 16 tasks with over 8,000 samples covering speech understanding, paralinguistic analysis, and cultural audio comprehension.
Persian poses unique audio understanding challenges through its classical poetry, traditional music, and pervasive code-switching - none captured by existing benchmarks. We introduce PARSA-Bench (Persian Audio Reasoning and Speech Assessment Benchmark), the first benchmark for evaluating large audio-language models on Persian language and culture, comprising 16 tasks and over 8,000 samples across speech understanding, paralinguistic analysis, and cultural audio understanding. Ten tasks are newly introduced, including poetry meter and style detection, traditional Persian music understanding, and code-switching detection. Text-only baselines consistently outperform audio counterparts, suggesting models may not leverage audio-specific information beyond what transcription alone provides. Culturally-grounded tasks expose a qualitatively distinct failure mode: all models perform near random chance on vazn detection regardless of scale, suggesting prosodic perception remains beyond the reach of current models. The dataset is publicly available at https://huggingface.co/datasets/MohammadJRanjbar/PARSA-Bench
Community
We introduce PARSA-Bench, the first benchmark for evaluating Large Audio-Language Models (LALMs) on Persian language and culture — 16 tasks, 8,000+ samples, across speech understanding, paralinguistic analysis, and Persian cultural audio understanding.
Key findings:
🔇 Audio processing is the dominant bottleneck — text-only baselines consistently outperform audio counterparts across all tasks
🎼 Poetry meter detection (vazn) is effectively unsolved at any scale — all models score near random chance, including GPT-4o and Gemini-2.5-Flash
🎙️ Poetry style (sabk) is the only task where audio beats text — vocal recitation carries genuine style-discriminative signal
📉 Proprietary scale offers no advantage on cultural tasks
🌍 Cultural audio understanding for non-Western languages like Persian remains a wide-open problem — current models lack the prosodic and cultural grounding needed, and no amount of scale fixes this
10 of the 16 tasks are newly introduced with no prior equivalent in any language, including Persian poetry meter/style detection, Dastgah music classification, and code-switching detection.
Dataset: https://huggingface.co/datasets/MohammadJRanjbar/PARSA-Bench
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation (2026)
- Voices of Civilizations: A Multilingual QA Benchmark for Global Music Understanding (2026)
- Covo-Audio Technical Report (2026)
- MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models (2026)
- DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models (2026)
- Eureka-Audio: Triggering Audio Intelligence in Compact Language Models (2026)
- TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper