Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models

🎉 Accepted at ICLR 2026

XModBench is a comprehensive benchmark designed to evaluate the cross-modal capabilities and consistency of omni-language models. It systematically assesses model performance across multiple modalities (text, vision, audio) and various cognitive tasks, revealing critical gaps in current state-of-the-art models.

• 🎯 Multi-Modal Evaluation: Comprehensive testing across text, vision, and audio modalities
• 🧩 5 Task Dimensions: Perception, Spatial, Temporal, Linguistic, and External Knowledge tasks
• 📊 13 SOTA Models Evaluated: Including Gemini 2.5 Pro, Qwen2.5-Omni, EchoInk-R1, and more
• 🔄 Consistency Analysis: Measures performance stability across different modal configurations
• 👥 Human Performance Baseline: Establishes human-level benchmarks for comparison

The dataset is available on Hugging Face: RyanWW/XModBench

Counts below reflect the actual released dataset (HF RyanWW/XModBench), summed over the 6 modality configurations.

The benchmark covers all six configurations of Audio, Vision (image or video), and Text as condition → answer options. Each configuration has the same 10,220 items (same semantics, permuted modality):

XModBench-Lite: a balanced 6,000-sample subset (5 families × 6 configs × 200) for fast evaluation.

XModBench/
├── benchmark/
│   ├── Data/                        # Raw media files (audio, image, video)
│   │   ├── vggss_audio_bench/       #   VGGSound audio clips
│   │   ├── landscape_audiobench/    #   Landscape images
│   │   ├── emotions/                #   Emotion classification media
│   │   └── ...
│   ├── tasks/                       # Source QA JSON files, organised by subtask
│   │   ├── 01_perception/
│   │   │   ├── finegrained/         #   6 modality-combo JSON files, 1000 instances each
│   │   │   ├── general_activities/
│   │   │   ├── instruments/
│   │   │   ├── instruments_comp/
│   │   │   └── natures/
│   │   ├── 02_spatial/
│   │   ├── 03_speech/
│   │   ├── 04_temporal/
│   │   └── 05_Exteral/
│   └── results/                     # Model evaluation results
├── models/                          # ★ per-model eval scripts (run.py …)
│   ├── Qwen2.5-Omni/  Qwen3-Omni/  Qwen2.5-VL/  OmniVinci/  VITA/  …
│   └── ...                          # upstream weights/impl install separately
└── scripts/                         # data-processing helpers (process/, download/)

Per-model evaluation code lives in models/ — each models/<Model>/run.py loads the benchmark, builds prompts, calls the model and scores. Only the XModBench-side scripts are tracked here; the upstream model weights/implementations are installed separately (see each model's upstream repo). For a turnkey reproducible path use the lmms-eval port.

#!/bin/bash
#SBATCH --job-name=VLM_eval        
#SBATCH --output=log/job_%j.out
#SBATCH --error=log/job_%j.log                        
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=4

echo "Running on host: $(hostname)"
echo "CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES"

module load conda
# conda activate vlm
conda activate omni

export audioBench='/home/xwang378/scratch/2025/AudioBench'

# python $audioBench/scripts/run.py \
#     --model gemini \
#     --task_name perception/vggss_audio_vision \
#     --sample 1000


# python $audioBench/scripts/run.py \
#     --model gemini \
#     --task_name perception/vggss_vision_audio \
#     --sample 1000

# python $audioBench/scripts/run.py \
#     --model gemini \
#     --task_name perception/vggss_vision_text \
#     --sample 1000

# python $audioBench/scripts/run.py \
#     --model gemini \
#     --task_name perception/vggss_audio_text \
#     --sample 1000

# Qwen2.5-Omni

# python $audioBench/scripts/run.py \
#         --model qwen2.5_omni \
#         --task_name perception/vggss_audio_text \
#         --sample 1000

python $audioBench/scripts/run.py \
        --model qwen2.5_omni \
        --task_name perception/vggss_vision_text \
        --sample 1000

We provide a fully reproducible evaluation path through lmms-eval (fork with XModBench tasks pre-integrated). The dataset is auto-downloaded from the HF Hub — no manual data prep.

Why dedicated model wrappers? Each XModBench item places media in both the question stem and every answer option (up to 5 media per item). lmms-eval's simple model interface only attaches one media object per request, so omni models would silently see just the first media. We therefore add chat-style *_interleave wrappers that feed the full interleaved prompt to the model. No upstream model file is modified.

git clone https://github.com/XingruiWang/lmms-eval.git
cd lmms-eval
pip install -e ".[all]"

python -m lmms_eval \
    --model qwen2_5_omni_interleave \
    --model_args pretrained=Qwen/Qwen2.5-Omni-7B,device_map=auto,attn_implementation=flash_attention_2 \
    --tasks xmod_bench_lite_a2t \
    --batch_size 1 --limit 8 --log_samples \
    --output_path logs/debug

submit_lite.sh launches all 6 modality configs with a resource-aware GPU profile (no-video configs on 1 GPU, video configs on 4) so the full sweep fits one QoS allocation:

# Qwen2.5-Omni-7B
./submit_lite.sh qwen2_5_omni_interleave Qwen/Qwen2.5-Omni-7B qwenomni3

# Qwen3-Omni-30B-A3B (MoE; all configs need 4 GPU)
LIGHT_GRES=gpu:a5000:4 HEAVY_GRES=gpu:a5000:4 \
  ./submit_lite.sh qwen3_omni_interleave Qwen/Qwen3-Omni-30B-A3B-Instruct qwenomni3 \
  device_map=auto,attn_implementation=flash_attention_2

# Level-2 metrics (by-config / by-family / disparity / imbalance)
python lmms_eval/tasks/xmod_bench/summarize.py \
    --logs logs/xmod_bench_lite/results_qwen2_5_omni_interleave/

TASKS=(xmod_bench_audio_text xmod_bench_text_audio \
       xmod_bench_audio_image xmod_bench_image_audio \
       xmod_bench_image_text xmod_bench_text_image \
       xmod_bench_audio_video xmod_bench_text_video \
       xmod_bench_video_audio xmod_bench_video_text)

python -m lmms_eval \
    --model qwen2_5_omni_interleave \
    --model_args pretrained=Qwen/Qwen2.5-Omni-7B,device_map=auto,attn_implementation=flash_attention_2 \
    --tasks "${TASKS[$SLURM_ARRAY_TASK_ID]}" \
    --batch_size 1 --log_samples \
    --output_path logs/xmod_bench_full/results

By-config accuracy on XModBench-Lite via lmms-eval, vs. the paper's full-set numbers (Table 2). Δ = Lite − paper.

• Qwen2.5-Omni reproduces the paper within |Δ| < 5 on all 6 configurations on the lightweight 6k Lite split — confirming the lmms-eval port is faithful.
• Qwen3-Omni (released after the paper) is reported here for the first time, using the identical wrapper/code path.
• Full-set (61,320-sample) lmms-eval runs use the same wrappers via the Section 4 command; numbers are updated in lmms_eval/tasks/xmod_bench/RESULTS.md as runs complete.

Per-run logs include overall accuracy plus per-config / per-family / per-subtask breakdowns; summarize.py emits the 17 Level-2 numbers (6 by-config, 5 by-family, 3 modality-disparity, 3 directional-imbalance).

By-configuration accuracy (%) over the six modality directions; Avg. is the mean over the six (Table 2 of the paper).

Vision-only models are evaluated only on text↔vision configs: Qwen2.5-VL 67.4 Avg., InternVL-3.5 61.7 Avg. (omitted from the six-way table).

Balanced split (5 families × 6 configs × 200), evaluated through the lmms-eval port with interleaved-multimedia wrappers.

Qwen2.5-Omni matches its full-set paper numbers within 5 points on every configuration, confirming the port is faithful. Qwen3-Omni post-dates the paper (first reported here). OmniVinci runs on its single-media-condition configs; its 4-option configs hit VILA-internal limits (see RESULTS.md — note: that file lives in the lmms-eval repo). New runs update as they complete.

• Strong Performance: Perception and linguistic tasks (~75% for best models)
• Weak Performance: Spatial (50.1%) and temporal reasoning (60.8%)
• Performance Drop: 15-25 points decrease in spatial/temporal vs. perception tasks

• Audio vs. Text: 20-49 point performance drop
• Audio vs. Vision: 33-point average gap
• Vision vs. Text: ~15-point disparity
• Consistency: Best models show 10-12 point standard deviation

• Vision↔Text: 9-17 point gaps between directions
• Audio↔Text: 6-8 point asymmetries
• Root Cause: Training data imbalance favoring image-to-text over inverse directions

If you use XModBench in your research, please cite our paper:

@article{wang2025xmodbench,
  title={XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models},
  author={Wang, Xingrui and others},
  journal={arXiv preprint arXiv:2510.15148},
  year={2025}
}

This project is licensed under the MIT License - see the LICENSE file for details.

We thank all contributors and the research community for their valuable feedback and suggestions.

• Project Lead: Xingrui Wang
• Email: [xwang378@jhu.edu]
• Website: https://xingruiwang.github.io/projects/XModBench/

• Project Website
• Paper
• Leaderboard
• Documentation

• Release Huggingface data
• Release data processing code
• Release data evaluation code

Note: XModBench is actively maintained and regularly updated with new models and evaluation metrics. For the latest updates, please check our releases page.