An Exploration of Mamba for Speech Self-Supervised Models

This repository provides the official implementation of the paper "An Exploration of Mamba for Speech Self-Supervised Models".

Our research explores Mamba-based HuBERT models as a linear-time alternative to Transformer-based speech self-supervised learning (SSL) architectures. These models utilize the Selective State Space mechanism to enable efficient long-context processing and demonstrate competitive performance in both ASR fine-tuning and streaming scenarios.

Follow these steps to set up the environment, apply the necessary patches, and begin pre-training.

Create the required conda environment using the provided mamba.yml file:

conda env create -f mamba.yml
conda activate <your_env_name>

To ensure compatibility with our custom Mamba implementation, you must replace the default mamba_simple.py file in your installed mamba_ssm library with the version provided in this repository's patch/ directory:

cp patch/modules/mamba_simple.py /path/to/your/anaconda3/envs/<your_env_name>/lib/python3.X/site-packages/mamba_ssm/modules/mamba_simple.py

Navigate to the specific training directory and execute the pre-training script:

cd fairseq-mamba/
bash run_hubert_XXX.sh