Unfolded Recursive Expectation-Maximization Neural Network for Speaker Tracking

Abstract

We propose a deep unfolded Recursive EM (REM) network for robust tracking of a single moving speaker in mild reverberant environments. Unlike classical REM algorithms, which rely on fixed-step-size decay schedules, the proposed architecture learns an adaptive update policy by unfolding the iterative procedure into differentiable layers. We introduce a Step Size Network that leverages Feature-wise Linear Modulation (FiLM) and Positional Encoding (PE) to dynamically adjust the recursion weights based on temporal context and convergence state. Experimental results for tracking a single speaker under reverberant conditions demonstrate that the proposed unfolded network outperforms the classical Cappé and Moulines REM (CREM) baseline, which employs a spatial grid search to map the estimated centroids to physical positions. In the single-speaker tracking task, the proposed method achieves a lower Root Mean Square Error (RMSE) than the CREM baseline, highlighting its potential for dynamic acoustic scenarios.

Index Terms—  Sound source tracking; Unfolding neural network; Recursive Expectation-Maximization; Pair-wise relative phase ratio.

1 Introduction

Sound Source Localization (SSL) is a fundamental task in modern audio applications, ranging from autonomous robots to smart-home consumer electronics. In realistic environments, estimating the spatial coordinates of a source is significantly complicated by reverberation and ambient noise. Traditional methods, such as Steered Response Power with Phase Transform (SRP-PHAT), Multiple Signals Classification (MUSIC), and Maximum Likelihood Estimation (MLE) [7, 15], often struggle in these adverse conditions. This has motivated a shift toward Deep Neural Network (DNN)-based approaches, which offer superior robustness in complex acoustic scenes [2, 5].

However, purely data-driven DNNs are often criticized for their lack of interpretability. Algorithm unfolding (or unrolling) bridges this gap by casting the iterations of classical algorithms as differentiable network layers [4, 10]. In our previous work [14], we successfully employed an unfolded network to estimate static multi-speaker positions by integrating the batch Expectation-Maximization (EM) procedure—previously used for Pair-Wise Relative Phase Ratio (PRP) clustering [16, 9]—into an unrolled architecture. While effective for static sources, batch-oriented methods lack the temporal consistency required for dynamic speaker tracking.

To address moving-source scenarios, recursive variants of the EM algorithm, e.g., the algorithm proposed by Cappé and Moulines [1] (henceforth denoted the CREM algorithm), update parameters frame-by-frame using recursively accumulated sufficient statistics. Several studies have integrated EM iterations into unfolding frameworks for image segmentation and clustering [3, 13]. Although recursive, a major limitation in the application of CREM is the selection of the step-size parameter $\gamma_{t}$, which governs the trade-off between past information and current observations. In static scenarios, a fixed decay schedule should be used. However, in tracking scenarios, where the speaker position changes over time, the step size should adapt to the underlying dynamics and is therefore often chosen empirically.

In this paper, we introduce a deep Unfolded CREM network embedded within an encoder-decoder architecture that learns an adaptive, data-driven step-size policy. By incorporating a conditioning sub-network based on FiLM [12] and sinusoidal PE [17], our model dynamically adjusts the recursion weights based on the current tracking state and temporal context. We demonstrate that this hybrid approach improves tracking accuracy and convergence speed compared to the classical recursive method.

2 Problem Formulation

Consider a single speaker moving in a reverberant enclosure, whose activity is captured by an array of $M$ microphone pairs. We assume the speaker moves at a constant velocity along either a linear or a circular path. The analysis is carried out in the Short-Time Fourier transform (STFT) domain. The signal captured at the $j$th microphone of the pair $m$, where $j=1,2$ and $m=1,\ldots,M$, at time-frame $t$ and frequency bin $k$ is modeled as:

$$z_{m,j}(t,k)=a_{m,j}(t,k)\cdot s(t,k)+n_{m,j}(t,k)$$ (1)

where $s(t,k)$ is the source signal, $n_{m,j}(t,k)$ is additive noise, and $a_{m,j}(t,k)$ are the Acoustic Transfer Functions relating the source positions and the microphone positions. The ATFs vary over time as the speaker moves along its trajectory $\mathbf{p}(t)$. Similar to our previous work [16, 14], we employ the PRP as the localization feature to improve robustness to signal-power variations. The observed PRP for pair $m$ is defined as:

$$\phi_{m}(t,k)\triangleq\frac{z_{m,2}(t,k)}{z_{m,1}(t,k)}\cdot\frac{|z_{m,1}(t,k)|}{|z_{m,2}(t,k)|},$$ (2)

and the observation vector at each time-frequency bin is

$$\boldsymbol{\phi}(t,k)=\left[\phi_{1}(t,k),\dots,\phi_{M}(t,k)\right]^{\top}.$$ (3)

To model the received PRP, we adopt a Complex Gaussian Mixture Model (CGMM) with two components ($C=2$): one associated with the moving speaker ($c=1$) and the other with noise or spurious estimates ($c=2$). The Probability Density Function (p.d.f.) for a single observation $\boldsymbol{\phi}(t,k)$ is given by:

$$f(\boldsymbol{\phi}(t,k))=\sum_{c=1}^{2}\psi_{c}(t)\cdot\mathcal{N}^{c}\left(\boldsymbol{\phi}(t,k);\tilde{\boldsymbol{\phi}}^{k}(\mathbf{p}_{c}(t)),\mathbf{\Sigma}_{c}\right)$$ (4)

where $\psi_{c}(t)$ is the prior probability of the $c$-th cluster. For the speaker cluster ($c=1$), the mean $\tilde{\boldsymbol{\phi}}^{k}(\mathbf{p}(t))$ should be the expected PRP analytically derived from the current position $\mathbf{p}(t)$, where each element $m$ is calculated as:

$$\tilde{\phi}_{m}^{k}(\mathbf{p}(t))\triangleq\exp\left(-j\frac{2\pi k}{K}\frac{|\mathbf{p}(t)-\mathbf{p}_{m,2}|-|\mathbf{p}(t)-\mathbf{p}_{m,1}|}{v\cdot T_{s}}\right)$$ (5)

where $v$ represents the sound velocity, and $T_{s}$ the sampling time of the continuous-time signal. For the noise cluster ($c=2$), the mean $\tilde{\boldsymbol{\phi}}^{k}(\mathbf{p}(t))$ does not originate from a specific point source, and hence, we expect this cluster to have a larger variance compared to the speaker’s variance.

Following [16], we assume independence of the PRP features across microphone pairs. We further simplify each covariance matrix to a scaled identity matrix, i.e., $\mathbf{\Sigma}_{c}=\sigma_{c}^{2}\mathbf{I}_{M}$. Finally, the p.d.f. of the entire observation set is given by

$$f(\boldsymbol{\phi};\boldsymbol{\theta}(t))=\\ \prod_{t,k}\left[\sum_{c=1}^{2}\psi_{c}(t)\prod_{m=1}^{M}\mathcal{N}^{c}\left(\phi_{m}(t,k);\tilde{\phi}_{m}^{k}(\mathbf{p}_{c}(t)),\sigma_{c}^{2}(t)\right)\right],$$ (6)

where $\boldsymbol{\theta}(t)$ denotes the set of time-varying parameters, comprising the Gaussian means, the priors, and variances. In the following section, we derive the REM algorithm, which updates $\hat{\boldsymbol{\theta}}(t)$ by accumulating the score of the log-likelihood over time, and demonstrate how this procedure is unfolded into a deep neural network to learn optimal, time-adaptive update steps.

3 Proposed Method

For online tracking of a moving speaker, we develop a recursive framework based on the CREM algorithm, which we subsequently unfold into a deep neural network.

The EM Algorithm: The EM algorithm requires defining three components and their probability models: the observations ($\boldsymbol{\phi}$), the target parameters ($\boldsymbol{\theta}$), and the hidden data ($\mathbf{x}$). In our case, we define the hidden data $\mathbf{x}$ to be the association of each Time-Frequency (TF) bin with either the source at a particular location or the background noise. Given the hidden data, the probability log-likelihood function of the observations is given by:

$$f(\mathbf{x},\boldsymbol{\phi};\boldsymbol{\theta})=\\ \prod_{t,k}\sum_{c}\psi_{c}\cdot x(t,k,c)\prod_{m}\mathcal{N}^{c}\left(\phi_{m}(t,k);\tilde{\phi}_{m}^{k}(\mathbf{p_{c}}),\sigma_{c}^{2}\right).$$ (7)

The E-step is therefore given by:

$$Q(\boldsymbol{\theta}|\boldsymbol{\theta}^{(\ell-1)})\triangleq E\left\{\log f(\mathbf{x},\boldsymbol{\phi};\boldsymbol{\theta})\Big|\boldsymbol{\phi};\boldsymbol{\theta}^{(\ell-1)}\right\}.$$ (8)

For implementing the E-step, it is sufficient to evaluate $\mu^{(\ell-1)}(t,k,c)\triangleq E\{x(t,k,c)|\boldsymbol{\phi}(t,k);\boldsymbol{\theta}^{(\ell-1)}\}$, given by:

$$\frac{\psi_{c}^{(\ell-1)}\prod_{m}\mathcal{N}^{c}\left(\phi_{m}(t,k);\tilde{\phi}_{m}^{k}(\mathbf{p}_{c})^{(\ell-1)},(\sigma_{c}^{2})^{(\ell-1)}\right)}{\sum_{c}\psi_{c}^{(\ell-1)}\prod_{m}\mathcal{N}^{c}\left(\phi_{m}(t,k);\tilde{\phi}_{m}^{k}(\mathbf{p}_{c})^{(\ell-1)},(\sigma_{c}^{2})^{(\ell-1)}\right)}.$$ (9)

Maximizing (9) with respect to the parameters $\boldsymbol{\theta}^{(\ell)}$ constitutes the M-step. We stress that this batch formulation assumes static speakers and operates entirely offline, making it unsuitable for real-time tracking in dynamic, time-varying environments.

The CREM Algorithm: The CREM algorithm adapts the batch EM procedure to an online scenario by substituting the iteration index with the time-frame index $t$. The core idea is to recursively estimate the auxiliary function, denoted $Q_{R}(\boldsymbol{\theta}|\boldsymbol{\theta}^{(t)})$, using a stochastic approximation step:

$$Q_{R}(\boldsymbol{\theta}|\boldsymbol{\theta}^{(t)})=(1-\gamma_{t})Q_{R}(\boldsymbol{\theta}|\boldsymbol{\theta}^{(t-1)})+\gamma_{t}Q(\boldsymbol{\theta}|\boldsymbol{\theta}^{(t)}),$$ (10)

where $\gamma_{t}$ is a time-varying smoothing coefficient (step-size) that determines the balance between past information and the current observation. The complete data log-likelihood is given by:

$$\log f(\mathbf{x}_{t},\boldsymbol{\phi}_{t};\boldsymbol{\theta})=\sum_{k,c}x(t,k,c)\cdot\\ \left(\log\psi_{c}(t)+\sum_{m}\log\mathcal{N}^{c}\left(\phi_{m}(t,k);\tilde{\phi}_{m}^{k}(\mathbf{p}_{c}(t)),\sigma_{c}^{2}(t)\right)\right).$$ (11)

Given that our CGMM likelihood (11) belongs to the exponential family, the maximization of $Q_{R}$ can be simplified by maintaining a recursive estimate of the sufficient statistics vector, $\boldsymbol{\eta}^{R}(t)$. The update rule for the sufficient statistics at frame $t$ is:

$$\boldsymbol{\eta}^{R}(t)=\boldsymbol{\eta}^{R}(t-1)+\gamma_{t}\left[\bar{\boldsymbol{\eta}}(t)-\boldsymbol{\eta}^{R}(t-1)\right],$$ (12)

where $\bar{\boldsymbol{\eta}}(t)$ is the expected sufficient statistics computed in the E-step, based on the posterior probability $\mu(t,k,c)$ that the $k$-th frequency bin at time $t$ belongs to cluster $c$. The term $\langle\boldsymbol{\overline{\eta}}(\mathbf{x}(t+1),\boldsymbol{\phi}(t+1)),\boldsymbol{\epsilon}(\boldsymbol{\theta})\rangle$, required for the E-step, can be simplified by calculating:

$$\mu(t+1,k,c)=E\{x(t+1,k,c)|\boldsymbol{\phi}_{m}(t+1,k);\boldsymbol{\theta}_{R}(t)\}$$ (13)

which is readily given by (9). Using this definition, the expected sufficient statistic $\bar{\boldsymbol{\eta}}(\boldsymbol{\mu}(t),\boldsymbol{\phi}(t))$ can be written as:

$$\bar{\boldsymbol{\eta}}(\boldsymbol{\mu}(t),\boldsymbol{\phi}(t))=\begin{pmatrix}\boldsymbol{\mu}(t)\\ \boldsymbol{\mu}(t)\odot|\boldsymbol{\phi}(t)|^{2}\\ \boldsymbol{\mu}(t)\odot\boldsymbol{\phi}^{*}(t)\\ \boldsymbol{\mu}(t)\odot\boldsymbol{\phi}(t)\end{pmatrix}$$ (14)

where the Hadamard product $\odot$ is applied with implicit broadcasting; specifically, when a dimension is absent in one of the tensors, it is replicated along that dimension to match the size of the other, followed by element-wise multiplication. The recursive sufficient statistic $\boldsymbol{\eta}^{R}(\boldsymbol{\mu}(t+1),\boldsymbol{\phi}(t+1))$ is defined as:

$$\boldsymbol{\eta}^{R}(\boldsymbol{\mu}(t+1),\boldsymbol{\phi}(t+1))\triangleq\begin{pmatrix}\boldsymbol{\eta}_{1}^{R}(t+1)\\ \boldsymbol{\eta}_{2}^{R}(t+1)\\ \boldsymbol{\eta}_{3}^{R}(t+1)\\ \boldsymbol{\eta}_{4}^{R}(t+1)\end{pmatrix}$$ (15)

where the updates are:

$$\boldsymbol{\eta}_{1}^{R}(t+1) =\boldsymbol{\eta}_{1}^{R}(t)+\gamma_{t+1}(\boldsymbol{\mu}(t+1)-\boldsymbol{\eta}_{1}^{R}(t))$$ (16a)

$$\boldsymbol{\eta}_{2}^{R}(t+1) =\boldsymbol{\eta}_{2}^{R}(t)+\gamma_{t+1}(\boldsymbol{\mu}(t+1)\odot|\boldsymbol{\phi}(t)|^{2}-\boldsymbol{\eta}_{2}^{R}(t))$$ (16b)

$$\boldsymbol{\eta}_{3}^{R}(t+1) =\boldsymbol{\eta}_{3}^{R}(t)+\gamma_{t+1}(\boldsymbol{\mu}(t+1)\odot\boldsymbol{\phi}^{*}(t)-\boldsymbol{\eta}_{3}^{R}(t))$$ (16c)

$$\boldsymbol{\eta}_{4}^{R}(t+1) =\boldsymbol{\eta}_{4}^{R}(t)+\gamma_{t+1}(\boldsymbol{\mu}(t+1)\odot\boldsymbol{\phi}(t)-\boldsymbol{\eta}_{4}^{R}(t)).$$ (16d)

The maximization step in the REM is similar to the maximization step in the batch EM. The M-step $\boldsymbol{\theta}_{c,R}(t+1)=\operatorname*{arg\,max}_{\boldsymbol{\theta_{c}}}\langle\boldsymbol{\eta}^{R}(\boldsymbol{\mu}(t+1),\boldsymbol{\phi}(t+1)),\boldsymbol{\epsilon}(\boldsymbol{\theta_{c}})\rangle$ yields the following parameter updates:

$$\psi_{c}(t+1)=\frac{1}{K}\sum_{k}\mu(t+1,k,c).$$ (17)

The M-step and the E-step can be merged into one recursion:

$$\boldsymbol{\psi}_{R}(t)=\frac{1}{K}\sum_{k}\boldsymbol{\eta}_{1}^{R}(t)$$ (18a)

and for the other parameters:

$$\tilde{\boldsymbol{\phi}}_{R}(\mathbf{p}(t))=\boldsymbol{\eta}_{4}^{R}(t)\oslash\boldsymbol{\eta}_{1}^{R}(t)$$ (18b)

$$\begin{aligned} \boldsymbol{\sigma}_{R}^{2}(t)&=\frac{1}{M\cdot\sum_{k}\boldsymbol{\eta}_{1}^{R}(t,k)}\sum_{k,m}\Big(\boldsymbol{\eta}_{1}^{R}(t,k)\odot|\tilde{\boldsymbol{\phi}}_{m}^{k}(\mathbf{p}(t))|^{2}\\ &\qquad+\boldsymbol{\eta}_{2}^{R}(t,k,m)-\boldsymbol{\eta}_{3}^{R}(t,k,m)\cdot\tilde{\boldsymbol{\phi}}_{m}^{k}(\mathbf{p}(t))\\ &\qquad-\boldsymbol{\eta}_{4}^{R}(t,k,m)\cdot\tilde{\boldsymbol{\phi^{*}}}_{m}^{k}(\mathbf{p}(t))\Big),\end{aligned}$$ (18c)

with $\oslash$ is an element-wise division operation. To infer the speaker’s physical 2D position in Cartesian coordinates, denoted as $\hat{\mathbf{p}}(t)$, from the recursively updated parameters, a spatial grid search is performed at each time step. The estimated position is obtained by minimizing the squared error between the estimated PRP and the theoretical PRP:

$$\hat{\mathbf{p}}(t)=\arg\min_{\mathbf{p}}\sum_{k,m}\left|\tilde{\phi}_{R,m}^{k}(\mathbf{p}(t))-\tilde{\phi}_{m}^{k}(\mathbf{p})\right|^{2},$$ (19)

where $\tilde{\phi}_{m}^{k}(\mathbf{p})$ is the theoretical value for a candidate location $\mathbf{p}$ as defined in (5).

The performance of the CREM algorithm is highly dependent on the choice of the step size $\gamma_{t}$. Classic stochastic approximation results guarantee convergence to a (static) local optimum given $\sum\gamma_{t}=\infty$ and $\sum\gamma_{t}^{2}<\infty$. A common schedule is $\gamma_{t}=(t+2)^{-\alpha}$, where $0.5<\alpha\leq 1$ balances the trade-off between convergence speed (smaller $\alpha$) and estimation stability (larger $\alpha$). To further enhance stability, we can adopt a mini-batch approach of size $B$, which interpolates between pure online ($B=1$) and batch ($B=T$) estimation [8]. However, since fixed schedules struggle with dynamic trajectories, we propose an unfolded architecture that learns an adaptive, data-driven step size, as described in the next section.

Unfolding CREM Neural Network: Building upon the interpretable Batch-EM unfolding framework presented in our previous work [14], we propose a deep unfolded architecture for the CREM algorithm. The network embeds the recursive EM steps within an encoder-decoder structure and introduces a data-driven mechanism to learn the optimal time-varying step size $\gamma_{t}$ (Figure 1).

(a) Full Unfolded CREM Architecture

(b) Step-Size Network

Fig. 1: Overview of the proposed tracking framework: (a) shows the unrolled EM layers within the encoder-decoder structure, and (b) details the FiLM-based adaptive step-size generation.

a) Encoder-decoder framework: The core of the architecture consists of an encoder and decoder. The encoder, implemented as a Fully Connected (FC) network, maps candidate spatial positions to an initial PRP vector. The decoder performs the inverse mapping, translating the refined PRP estimates back to spatial coordinates $\hat{\mathbf{p}}(t)$. By using the decoder at each time step, the network provides online tracking of the speaker’s trajectory. To enhance stability, each layer processes mini-batches of size $B$ by interpolating between the online and batch regimes.

b) Dynamic step-size learning via FiLM and PE: The primary innovation in this architecture is the Step-Size Network, which replaces the heuristic schedules of $\gamma_{t}$ with a learned, adaptive function. Since the optimal update step depends on both the current convergence state and the temporal context, we employ FiLM [12] and PE [17]. To provide the network with information about the current iteration, we use a sinusoidal positional encoding, $\textrm{PE}(t)$.The step-size network receives a conditioning vector $\mathbf{v}_{t}=[\textrm{PE}(t),\boldsymbol{\psi}(t-1),\boldsymbol{\sigma}^{2}(t-1),\boldsymbol{\phi}(t-1)]^{\top}$, which is processed through a Layer-Normalized Multi-Layer Perceptron (MLP) to generate the FiLM parameters, $\alpha$ (scaling) and $\beta$ (shifting). The intermediate features are extracted from the concatenation of the current observation $\boldsymbol{\phi}(t)$ and the temporal difference between successive observations $\Delta\boldsymbol{\phi}(t)=\boldsymbol{\phi}(t)-\boldsymbol{\phi}(t-1)$. Then, the FiLM parameters modulate the intermediate features to produce the scalar step size $\gamma_{t}$ used in (12). The architecture of the step-size network is shown in Figure 1.

c) Weighted loss: The network is trained using a time-weighted composite loss function. To account for the initial convergence period of the REM, we introduce a warmup weight $w(t)$ that scales the loss at each time step. The spatial accuracy is governed by a logarithmic distance error, which provides a sub-linear penalty to stabilize gradients against large initial offsets:

$$\begin{split}\mathcal{L}=&\frac{1}{T}\sum_{t=0}^{T-1}w(t)\Big[(1-\lambda)\log(1+|\hat{\mathbf{p}}(t)-\mathbf{p}_{\text{true}}(t)|^{2})\\ &+\lambda\cdot(1-\text{CosSim}(\tilde{\boldsymbol{\phi}}(\mathbf{p}(t)),\tilde{\boldsymbol{\phi}}(\mathbf{p}_{\text{true}}(t)))\Big]\end{split}$$ (20)

where $\text{CosSim}(\cdot,\cdot)$ denotes the cosine similarity operator.

4 Experimental Study

Data Generation and Setup: We evaluated the proposed unfolded network using a synthetic dataset derived from the Wall Street Journal (WSJ) corpus [11]. The simulations were implemented in Python using the image-method-based signal generator [6]. We simulated a single moving speaker in rectangular rooms with random dimensions (height: 2.2–2.6 m; width/length: 5–7 m) with eight microphone pairs (intra-pair distance: 0.2 m). The speaker moves at a constant velocity of $0.5$ m/s, following either a linear trajectory or a circular path with a radius ranging from $1$ to $3$ m. Acoustic conditions include either anechoic or reverberant environments with $\textrm{RT}_{60}=0.3$ s and additive white noise at $\text{SNR}=30$ dB. The dataset consists of $8{,}000$ training and $2{,}000$ validation samples.

Signal Processing and Data Flow: The input signals, sampled at $16$ kHz, feature durations of $5$ seconds in anechoic scenarios and $9$ seconds in reverberant conditions. Transformed into the TF domain using an STFT (window size of $1024$, hop size of $512$ samples), these configurations yield $157$ and $282$ time frames, respectively. To optimize efficiency, only frequency bins within the $500-1500$ Hz range are considered. Tracking is performed with a mini-batch size of $B=10$ consecutive time frames at each step, employing a sliding window with a step size of $3$ frames between consecutive CREM updates to maintain temporal continuity.

Encoder and Decoder Architectures: The FC encoder serves as the initialization block, mapping random room coordinates to the initial PRP vector through $7$ layers with increasing widths: $[6,48,192,384,768,1536,2080]$. Each layer is followed by LayerNorm and Rectified Linear Unit (ReLU) activation, while the final output uses a tanh activation to ensure the stability of the PRP values. The decoder, which maps the refined PRP back to spatial coordinates, comprises $6$ layers with decreasing dimensions: $[1040,512,256,128,64,2]$. All layers utilize ReLU activation, except for the final output layer, which provides the continuous $2$D coordinates.

Step-Size Network with FiLM: The adaptive step size $\gamma_{t}$ is estimated using a multi-branch conditioning network. (i) Parameter Branch: $\boldsymbol{\psi}(t)$ and $\boldsymbol{\sigma}^{2}(t)$ are mapped to a $16$-dimensional feature; (ii) Manifold Branch: The estimated $\tilde{\boldsymbol{\phi}}(\mathbf{p}(t))$ passes through $4$ layers ($[512,256,128,64]$) with ReLU; (iii) Temporal Branch: The time index $t$ is encoded via a $32$-dimensional PE. These three branches are concatenated and passed through LayerNorm to generate the FiLM parameters ($\alpha,\beta$). Two separate generators are used: a $3$-layer FC ($[256,512,1024]$) for a $512$-dim modulation, and a $2$-layer FC ($[164,256]$) for a $128$-dimensional modulation.

The main step-size estimation path concatenates the current observation $\boldsymbol{\phi}(t)$ and the temporal difference $\Delta\boldsymbol{\phi}(t)$. This input is processed through a $512$-dimensional FC layer, followed by the first FiLM block, two FC layers ($[256,128]$), the second FiLM block, and finally $3$ FC layers ($[64,32,1]$). A sigmoid activation is applied to the final scalar output to bound the learned step size $\gamma_{t}\in[0,1]$.

Training and Stability: Training was performed using the composite loss in (20) with $\lambda=0.5$. To handle the initial convergence period, we employed a linear warmup schedule for the loss weights $w(t)$, increasing from $0.1$ to $1.0$ over the first $20$ time steps and remaining constant thereafter. To ensure numerical stability during the unfolding of recursive updates, we applied several regularization techniques: (i) gradient clipping with a threshold of $1.5$; (ii) weight decay of $10^{-4}$; (iii) a clamp value of $50$ on the estimated variances $\sigma^{2}$; and (iv) a small epsilon $\epsilon=10^{-8}$ added to all denominators in the CREM update rules. The network was optimized using Adam with an initial learning rate of $10^{-3}$.

Results and Discussion: We evaluated the proposed approach against the CREM baseline using $100$ unseen speech signals. To demonstrate the impact of our learned model, the baseline performance was tested across multiple fixed step sizes, specifically evaluating $\gamma_{t}\in\{0.25,0.5,0.75\}\ \forall\,t$, coupled with a spatial grid search. These evaluation scenarios incorporated diverse room sizes and varying trajectory locations to assess localization accuracy, tracking adaptability, and robustness across different acoustic conditions. autoreftab:results summarizes the results. We report the warmup-weighted average RMSE between the estimated and ground-truth speaker positions, along with the percentage of test trajectories for which the time-averaged positional error exceeds $0.5$ m.

Table 1: Average Localization Performance: Unfolded Network vs. CREM Baseline.

Rever. Level	Unfolded CREM Network		CREM Baseline ($\gamma=[0.25,0.5,0.75]$)
	RMSE (m)	>0.5 m (%)	RMSE (m)	>0.5 m (%)
$T_{60}=0$ s	0.25	3	[0.28, 0.67, 1.44]	[1, 29, 61]
$T_{60}=0.3$ s	0.55	56	[0.74, 0.81, 0.86]	[87, 93, 97]

Under anechoic conditions, the unfolded network outperforms the baselines with larger fixed step sizes while achieving performance comparable to the $\gamma=0.25$ baseline. This is consistent with the learned step size dynamically converging to a similar low value. In the more challenging reverberant scenario ($T_{60}=0.3$ s), the proposed method continues to outperform all fixed-step baselines, although its performance degrades relative to the anechoic case. The advantage over the $\gamma=0.5$ baseline is further illustrated by the error distribution in Fig. 2, where the unfolded network (blue) exhibits a substantially higher concentration of trajectories with low RMSE than the baseline (orange).

Fig. 2: Comparison of localization RMSE distribution between the unfolded network and the CREM baseline ($\gamma_{t}=0.5$). The proposed method’s error distribution is more concentrated at lower values; median (solid) and mean (dashed) lines are provided to illustrate the performance gap.

(a) Speaker tracking in an anechoic room ($\textrm{RT}_{60}=0$ s). Average RMSE $=0.09$ m

(b) Speaker tracking in a reverberant room ($\textrm{RT}_{60}=0.3$ s). Average RMSE $=0.57$ m

Fig. 3: Trajectory estimation examples from test data. True positions are marked by dots, and network estimates by ‘x‘ markers. The color gradient represents the progression of time and the accompanying plot displays the corresponding learned step-size values.

The network’s tracking capabilities are further qualitatively illustrated in Fig. 3, which shows the estimated trajectories under both anechoic and reverberant conditions.

5 Conclusion

We presented an unfolded REM network for speaker tracking that replaces fixed step-size schedules with a learned, adaptive update policy. Experimental results demonstrate that the proposed model outperforms the classical CREM baseline, achieving lower RMSE and improved stability in both anechoic and reverberant environments. Analysis of the learned parameters shows that the step size $\gamma_{t}$ converges to a distinct range for each acoustic setting. In anechoic scenarios, $\gamma_{t}$ fluctuates around $0.25$ (peaking at $0.55$), whereas in reverberant conditions it decreases to $0.02$–$0.1$. The network learns to stabilize the step size, likely because the speaker velocity remains constant. Future work will focus on improving robustness under highly reverberant conditions and extending the model to more complex scenarios with variable speaker velocities and trajectories.