Spatial-Omni

Checkpoints for Spatial-Omni: a dedicated SO-Encoder adds first-order ambisonic (FOA) audio to audio-language models and Omni models alongside their native audio encoder.

Paper · Code · SO-Dataset · SO-Bench

Models

Family	Base model	Checkpoints	Implementation
SO-7B	Qwen/Qwen2.5-Omni-7B	SO / MIX	Code
SO-30B	Qwen/Qwen3-Omni-30B-A3B-Instruct	SO / MIX	Code
SO-4B	microsoft/Phi-4-multimodal-instruct	SO / MIX	Code
SO-AF3	nvidia/audio-flamingo-3	SO / MIX	Code

The shared SO-Encoder is provided separately. Adaptation files contain the trained encoder, projector and language LoRA parameters. SO-30B includes its trained MoE routers; MIX files also include learned null tokens. Load each adaptation with its matching base model and code branch.

Usage

Download the encoder and the model family you need, for example:

hf download dieKarotte/Spatial-Omni \
  --include 'SO-Encoder/**' 'SO-30B/**' \
  --local-dir checkpoints/Spatial-Omni

Keep each checkpoint beside its train_args.json and release_config.json. Use the corresponding code branch's inference or training command with explicit local base-model and SO-Encoder paths.

Input follows the dataset's four-channel FOA convention at 16 kHz, with W in channel 0. Preserve channel order. The spatial branch uses a 20-second window, 10 Hz encoder features and 2.5 projected spatial tokens per second.

For Phi-4, keep --merge-speech-lora consistent with these checkpoints. AF3 requires the base repository's native sound tower, sound projector and language model. The 30B loader also restores the trained routers. Each branch documents its supported mono inference options.

Recorded evaluations

SO-Bench uses 7,877 questions, MMAU test-mini 1,000, and MMAU-Pro 4,163 unique questions.

SO-Bench follows the released scripts/score_sobench.py: gpt-4o-mini normalization, 20° azimuth, 10° elevation, 1 m distance and 0.2 s onset tolerances. The aggregate averages task-specific per-question values, including temporal IoU.

MIX training

MIX training use spatial:replay 3:1 interleaving. Their saved configurations describe the actual training runs.

License and citation

The Spatial-Omni checkpoints use CC BY-NC-SA 4.0. Base-model weights and third-party components retain their respective licenses; download them from the linked original repositories.

Please cite Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding, arXiv:2606.10738.

@article{zhu2026spatial,
  title={Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding},
  author={Zhu, Zhiyuan and Chen, Yixuan and Shao, Yiwen and Guo, Wenxiang and Pan, Changhao and Zhang, Yu and Wang, Yuxiang and Liu, Wei and Zhang, Houhua and Zeng, Chengkuan and others},
  journal={arXiv preprint arXiv:2606.10738},
  year={2026}
}