Sensory Measurements On each trial t t t
m A , l A ( t ) ′ ∼ N ( s A ′ , σ ′ A V , A 2 ) m V , l V ( t ) ′ ∼ N ( s V ′ , σ ′ A V , V 2 ) \begin{aligned}
m'_{A,l_A(t)} &\sim \mathcal{N}\!\bigl(s'_A,\; {\sigma'}^2_{AV,A}\bigr) \\[6pt]
m'_{V,l_V(t)} &\sim \mathcal{N}\!\bigl(s'_V,\; {\sigma'}^2_{AV,V}\bigr)
\end{aligned} m A , l A ( t ) ′ m V , l V ( t ) ′ ∼ N ( s A ′ , σ ′ A V , A 2 ) ∼ N ( s V ′ , σ ′ A V , V 2 ) where the stimulus locations in internal space after accumulated recalibration are:
s A ′ = a A s A + b A + Δ A i ( t ) s V ′ = s V + Δ V i ( t ) \begin{aligned}
s'_A &= a_A\, s_A + b_A + \Delta_{A_i}(t) \\[6pt]
s'_V &= s_V + \Delta_{V_i}(t)
\end{aligned} s A ′ s V ′ = a A s A + b A + Δ A i ( t ) = s V + Δ V i ( t ) Causal Inference We assume a flat supra-modal prior P ( s A V ′ ) P(s'_{AV}) P ( s A V ′ ) μ p ′ \mu'_p μ p ′ σ p ′ 2 {\sigma'_p}^2 σ p ′ 2
Common cause (C = 1 C = 1 C = 1 If the brain infers both cues originate from the same source, the location estimate is the reliability-weighted average of the two measurements and the supra-modal prior:
s ^ A i , l A ( t ) , C = 1 ′ = s ^ V i , l V ( t ) , C = 1 ′ = m A , l A ( t ) ′ σ ′ A V , A − 2 + m V , l V ( t ) ′ σ ′ A V , V − 2 + μ p ′ σ ′ p − 2 σ ′ A V , A − 2 + σ ′ A V , V − 2 + σ ′ p − 2 \hat{s}'_{A_i,l_A(t),C=1} = \hat{s}'_{V_i,l_V(t),C=1} = \frac{m'_{A,l_A(t)}{\sigma'}^{-2}_{AV,A} + m'_{V,l_V(t)}{\sigma'}^{-2}_{AV,V} + \mu'_p {\sigma'}^{-2}_p}{{\sigma'}^{-2}_{AV,A} + {\sigma'}^{-2}_{AV,V} + {\sigma'}^{-2}_p} s ^ A i , l A ( t ) , C = 1 ′ = s ^ V i , l V ( t ) , C = 1 ′ = σ ′ A V , A − 2 + σ ′ A V , V − 2 + σ ′ p − 2 m A , l A ( t ) ′ σ ′ A V , A − 2 + m V , l V ( t ) ′ σ ′ A V , V − 2 + μ p ′ σ ′ p − 2 Separate causes (C = 2 C = 2 C = 2 If the brain infers the cues originate from different sources, each modality's estimate is formed independently from its own measurement and the supra-modal prior:
s ^ A i , l A ( t ) , C = 2 ′ = m A , l A ( t ) ′ σ ′ A V , A − 2 + μ p ′ σ ′ p − 2 σ ′ A V , A − 2 + σ ′ p − 2 s ^ V i , l V ( t ) , C = 2 ′ = m V , l V ( t ) ′ σ ′ A V , V − 2 + μ p ′ σ ′ p − 2 σ ′ A V , V − 2 + σ ′ p − 2 \begin{aligned}
\hat{s}'_{A_i,l_A(t),C=2} &= \frac{m'_{A,l_A(t)}{\sigma'}^{-2}_{AV,A} + \mu'_p {\sigma'}^{-2}_p}{{\sigma'}^{-2}_{AV,A} + {\sigma'}^{-2}_p} \\[6pt]
\hat{s}'_{V_i,l_V(t),C=2} &= \frac{m'_{V,l_V(t)}{\sigma'}^{-2}_{AV,V} + \mu'_p {\sigma'}^{-2}_p}{{\sigma'}^{-2}_{AV,V} + {\sigma'}^{-2}_p}
\end{aligned} s ^ A i , l A ( t ) , C = 2 ′ s ^ V i , l V ( t ) , C = 2 ′ = σ ′ A V , A − 2 + σ ′ p − 2 m A , l A ( t ) ′ σ ′ A V , A − 2 + μ p ′ σ ′ p − 2 = σ ′ A V , V − 2 + σ ′ p − 2 m V , l V ( t ) ′ σ ′ A V , V − 2 + μ p ′ σ ′ p − 2 Final location estimates (model averaging) The final perceptual estimate for each modality is an average of the common cause and separate causes estimates, with each estimate weighted by the posterior probability of each causal structure:
s ^ A i , l A ( t ) ′ = s ^ A i , l A ( t ) , C = 1 ′ ⋅ P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) + s ^ A i , l A ( t ) , C = 2 ′ ⋅ ( 1 − P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) ) \begin{aligned}
\hat{s}'_{A_i,l_A(t)} &= \hat{s}'_{A_i,l_A(t),C=1} \cdot P(C\!=\!1 \mid m'_{A,l_A(t)}, m'_{V,l_V(t)}) \\
&\quad + \hat{s}'_{A_i,l_A(t),C=2} \cdot \bigl(1 - P(C\!=\!1 \mid m'_{A,l_A(t)}, m'_{V,l_V(t)})\bigr)
\end{aligned} s ^ A i , l A ( t ) ′ = s ^ A i , l A ( t ) , C = 1 ′ ⋅ P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) + s ^ A i , l A ( t ) , C = 2 ′ ⋅ ( 1 − P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) ) s ^ V i , l V ( t ) ′ = s ^ V i , l V ( t ) , C = 1 ′ ⋅ P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) + s ^ V i , l V ( t ) , C = 2 ′ ⋅ ( 1 − P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) ) \begin{aligned}
\hat{s}'_{V_i,l_V(t)} &= \hat{s}'_{V_i,l_V(t),C=1} \cdot P(C\!=\!1 \mid m'_{A,l_A(t)}, m'_{V,l_V(t)}) \\
&\quad + \hat{s}'_{V_i,l_V(t),C=2} \cdot \bigl(1 - P(C\!=\!1 \mid m'_{A,l_A(t)}, m'_{V,l_V(t)})\bigr)
\end{aligned} s ^ V i , l V ( t ) ′ = s ^ V i , l V ( t ) , C = 1 ′ ⋅ P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) + s ^ V i , l V ( t ) , C = 2 ′ ⋅ ( 1 − P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) ) Posterior probability of common cause P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) = P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 1 ) ⋅ P ( C = 1 ) P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 1 ) ⋅ P ( C = 1 ) + P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 2 ) ⋅ ( 1 − P ( C = 1 ) ) \begin{aligned}
&P(C = 1 \mid m'_{A,l_A(t)}, m'_{V,l_V(t)}) = \\
&\frac{P(m'_{A,l_A(t)}, m'_{V,l_V(t)} \mid C\!=\!1) \cdot P(C\!=\!1)}
{P(m'_{A,l_A(t)}, m'_{V,l_V(t)} \mid C\!=\!1) \cdot P(C\!=\!1) + P(m'_{A,l_A(t)}, m'_{V,l_V(t)} \mid C\!=\!2) \cdot (1 - P(C\!=\!1))}
\end{aligned} P ( C = 1 ∣ m A , l A ( t ) ′ , m V , l V ( t ) ′ ) = P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 1 ) ⋅ P ( C = 1 ) + P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 2 ) ⋅ ( 1 − P ( C = 1 )) P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 1 ) ⋅ P ( C = 1 ) Common cause likelihood P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 1 ) = ∫ P ( m A , l A ( t ) ′ ∣ s A V ′ ) P ( m V , l V ( t ) ′ ∣ s A V ′ ) P ( s A V ′ ) d s A V ′ = 1 2 π σ ′ A V , A 2 σ ′ A V , V 2 + σ ′ A V , A 2 σ ′ p 2 + σ ′ A V , V 2 σ ′ p 2 × exp [ − 1 2 ( m A , l A ( t ) ′ − m V , l V ( t ) ′ ) 2 σ ′ p 2 + ( m A , l A ( t ) ′ − μ p ′ ) 2 σ ′ A V , V 2 + ( m V , l V ( t ) ′ − μ p ′ ) 2 σ ′ A V , A 2 σ ′ A V , A 2 σ ′ A V , V 2 + σ ′ A V , A 2 σ ′ p 2 + σ ′ A V , V 2 σ ′ p 2 ] \begin{aligned}
&P(m'_{A,l_A(t)}, m'_{V,l_V(t)} \mid C\!=\!1) = \int P(m'_{A,l_A(t)} \mid s'_{AV})\, P(m'_{V,l_V(t)} \mid s'_{AV})\, P(s'_{AV}) \, ds'_{AV} \\[6pt]
&= \frac{1}{2\pi \sqrt{{\sigma'}^2_{AV,A}{\sigma'}^2_{AV,V} + {\sigma'}^2_{AV,A}{\sigma'}^2_{p} + {\sigma'}^2_{AV,V}{\sigma'}^2_{p}}} \\
&\quad \times \exp \!\left[ -\frac{1}{2} \frac{(m'_{A,l_A(t)} \!-\! m'_{V,l_V(t)})^2 {\sigma'}^2_{p} + (m'_{A,l_A(t)} \!-\! \mu'_p)^2 {\sigma'}^2_{AV,V} + (m'_{V,l_V(t)} \!-\! \mu'_p)^2 {\sigma'}^2_{AV,A}}{{\sigma'}^2_{AV,A}{\sigma'}^2_{AV,V} + {\sigma'}^2_{AV,A}{\sigma'}^2_{p} + {\sigma'}^2_{AV,V}{\sigma'}^2_{p}} \right]
\end{aligned} P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 1 ) = ∫ P ( m A , l A ( t ) ′ ∣ s A V ′ ) P ( m V , l V ( t ) ′ ∣ s A V ′ ) P ( s A V ′ ) d s A V ′ = 2 π σ ′ A V , A 2 σ ′ A V , V 2 + σ ′ A V , A 2 σ ′ p 2 + σ ′ A V , V 2 σ ′ p 2 1 × exp [ − 2 1 σ ′ A V , A 2 σ ′ A V , V 2 + σ ′ A V , A 2 σ ′ p 2 + σ ′ A V , V 2 σ ′ p 2 ( m A , l A ( t ) ′ − m V , l V ( t ) ′ ) 2 σ ′ p 2 + ( m A , l A ( t ) ′ − μ p ′ ) 2 σ ′ A V , V 2 + ( m V , l V ( t ) ′ − μ p ′ ) 2 σ ′ A V , A 2 ] Separate causes likelihood P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 2 ) = ( ∫ P ( m A , l A ( t ) ′ ∣ s A ′ ) P ( s A ′ ) d s A ′ ) ( ∫ P ( m V , l V ( t ) ′ ∣ s V ′ ) P ( s V ′ ) d s V ′ ) = 1 2 π ( σ ′ A V , A 2 + σ ′ p 2 ) ( σ ′ A V , V 2 + σ ′ p 2 ) × exp [ − 1 2 ( ( m A , l A ( t ) ′ − μ p ′ ) 2 σ ′ A V , A 2 + σ ′ p 2 + ( m V , l V ( t ) ′ − μ p ′ ) 2 σ ′ A V , V 2 + σ ′ p 2 ) ] \begin{aligned}
&P(m'_{A,l_A(t)}, m'_{V,l_V(t)} \mid C\!=\!2) = \!\left( \int P(m'_{A,l_A(t)} \mid s'_A)\, P(s'_A) \, ds'_A \right)\!\left( \int P(m'_{V,l_V(t)} \mid s'_V)\, P(s'_V) \, ds'_V \right) \\[6pt]
&= \frac{1}{2\pi \sqrt{({\sigma'}^2_{AV,A} + {\sigma'}^2_p)({\sigma'}^2_{AV,V} + {\sigma'}^2_p)}} \;\times\; \exp \!\left[ -\frac{1}{2} \left( \frac{(m'_{A,l_A(t)} - \mu'_p)^2}{{\sigma'}^2_{AV,A} + {\sigma'}^2_p} + \frac{(m'_{V,l_V(t)} - \mu'_p)^2}{{\sigma'}^2_{AV,V} + {\sigma'}^2_p} \right) \right]
\end{aligned} P ( m A , l A ( t ) ′ , m V , l V ( t ) ′ ∣ C = 2 ) = ( ∫ P ( m A , l A ( t ) ′ ∣ s A ′ ) P ( s A ′ ) d s A ′ ) ( ∫ P ( m V , l V ( t ) ′ ∣ s V ′ ) P ( s V ′ ) d s V ′ ) = 2 π ( σ ′ A V , A 2 + σ ′ p 2 ) ( σ ′ A V , V 2 + σ ′ p 2 ) 1 × exp [ − 2 1 ( σ ′ A V , A 2 + σ ′ p 2 ( m A , l A ( t ) ′ − μ p ′ ) 2 + σ ′ A V , V 2 + σ ′ p 2 ( m V , l V ( t ) ′ − μ p ′ ) 2 ) ] Recalibration Process The fixed-ratio model Δ A i ( t + 1 ) = Δ A i ( t ) + α A ⋅ ( m V , l V ( t ) ′ − m A , l A ( t ) ′ ) Δ V i ( t + 1 ) = Δ V i ( t ) + α V ⋅ ( m A , l A ( t ) ′ − m V , l V ( t ) ′ ) \begin{aligned}
\Delta_{A_i}(t+1) &= \Delta_{A_i}(t) + \alpha_A \cdot (m'_{V,l_V(t)} - m'_{A,l_A(t)}) \\[6pt]
\Delta_{V_i}(t+1) &= \Delta_{V_i}(t) + \alpha_V \cdot (m'_{A,l_A(t)} - m'_{V,l_V(t)})
\end{aligned} Δ A i ( t + 1 ) Δ V i ( t + 1 ) = Δ A i ( t ) + α A ⋅ ( m V , l V ( t ) ′ − m A , l A ( t ) ′ ) = Δ V i ( t ) + α V ⋅ ( m A , l A ( t ) ′ − m V , l V ( t ) ′ ) The supervised modality-specific model 1. For cued modality
If c ( t ) = A c(t) = A c ( t ) = A
e A i ( t ) = feedback A ( t ) − s ^ A i , l A ( t ) ′ e_{A_i}(t) = \text{feedback}_A(t) - \hat{s}'_{A_i,l_A(t)} e A i ( t ) = feedback A ( t ) − s ^ A i , l A ( t ) ′ If c ( t ) = V c(t) = V c ( t ) = V
e V i ( t ) = feedback V ( t ) − s ^ V i , l V ( t ) ′ e_{V_i}(t) = \text{feedback}_V(t) - \hat{s}'_{V_i,l_V(t)} e V i ( t ) = feedback V ( t ) − s ^ V i , l V ( t ) ′ Here, if large discrepancy → low P ( C = 1 ∣ ⋅ ) P(C\!=\!1 \mid \cdot) P ( C = 1 ∣ ⋅ ) s ^ ′ \hat{s}' s ^ ′
2. For uncued modality
The overall expression is:e V i ( t ) = s ^ V i , l V ( t ) ′ − m V , l V ( t ) ′ = [ P ( C = 1 ∣ ⋅ ) ⋅ s ^ V i , l V ( t ) , C = 1 ′ + ( 1 − P ( C = 1 ∣ ⋅ ) ) ⋅ s ^ V i , l V ( t ) , C = 2 ′ ] − m V , l V ( t ) ′ = P ( C = 1 ∣ ⋅ ) ⋅ ( s ^ V i , l V ( t ) , C = 1 ′ − m V , l V ( t ) ′ ) + ( 1 − P ( C = 1 ∣ ⋅ ) ) ⋅ ( s ^ V i , l V ( t ) , C = 2 ′ − m V , l V ( t ) ′ ) \begin{aligned}
e_{V_i}(t) &= \hat{s}'_{V_i,l_V(t)} - m'_{V,l_V(t)} \\[6pt]
&= \bigl[P(C\!=\!1 \mid \cdot) \cdot \hat{s}'_{V_i,l_V(t),C=1} + (1 - P(C\!=\!1 \mid \cdot)) \cdot \hat{s}'_{V_i,l_V(t),C=2}\bigr] - m'_{V,l_V(t)} \\[6pt]
&= P(C\!=\!1 \mid \cdot) \cdot \bigl(\hat{s}'_{V_i,l_V(t),C=1} - m'_{V,l_V(t)}\bigr) + (1 - P(C\!=\!1 \mid \cdot)) \cdot \bigl(\hat{s}'_{V_i,l_V(t),C=2} - m'_{V,l_V(t)}\bigr)
\end{aligned} e V i ( t ) = s ^ V i , l V ( t ) ′ − m V , l V ( t ) ′ = [ P ( C = 1 ∣ ⋅ ) ⋅ s ^ V i , l V ( t ) , C = 1 ′ + ( 1 − P ( C = 1 ∣ ⋅ )) ⋅ s ^ V i , l V ( t ) , C = 2 ′ ] − m V , l V ( t ) ′ = P ( C = 1 ∣ ⋅ ) ⋅ ( s ^ V i , l V ( t ) , C = 1 ′ − m V , l V ( t ) ′ ) + ( 1 − P ( C = 1 ∣ ⋅ )) ⋅ ( s ^ V i , l V ( t ) , C = 2 ′ − m V , l V ( t ) ′ ) 3. Shift update
Δ A i ( t + 1 ) = Δ A i ( t ) + α A ⋅ e A i ( t ) Δ V i ( t + 1 ) = Δ V i ( t ) + α V ⋅ e V i ( t ) \begin{aligned}
\Delta_{A_i}(t+1) &= \Delta_{A_i}(t) + \alpha_A \cdot e_{A_i}(t) \\[6pt]
\Delta_{V_i}(t+1) &= \Delta_{V_i}(t) + \alpha_V \cdot e_{V_i}(t)
\end{aligned} Δ A i ( t + 1 ) Δ V i ( t + 1 ) = Δ A i ( t ) + α A ⋅ e A i ( t ) = Δ V i ( t ) + α V ⋅ e V i ( t ) Shared parameters (modality-specific & combined)
sA (auditory location) 5.0
aA (gain) 1.00
bA (bias) 0.0
sV (visual location) 0.0
feedbackA (°) 5.0
σ'²AV,A (auditory var) 4.0
σ'²AV,V (visual var) 1.0
σ'²p (prior var) 200
μ'p (prior mean) 0.0
P(C=1) prior 0.50
αA (auditory learning rate) 0.030
αV (visual learning rate) 0.030
Number of trials 200
Cued modality:
Resample noise
The supervised combined model If cued modality is A A A
Under C = 1 C = 1 C = 1
e C = 1 ( t ) = feedback ( t ) − s ^ A i , l A ( t ) , C = 1 ′ e_{C=1}(t) = \text{feedback}(t) - \hat{s}'_{A_i,l_A(t),C=1} e C = 1 ( t ) = feedback ( t ) − s ^ A i , l A ( t ) , C = 1 ′ where s ^ A i , l A ( t ) , C = 1 ′ = s ^ V i , l V ( t ) , C = 1 ′ \hat{s}'_{A_i,l_A(t),C=1} = \hat{s}'_{V_i,l_V(t),C=1} s ^ A i , l A ( t ) , C = 1 ′ = s ^ V i , l V ( t ) , C = 1 ′
Under C = 2 C = 2 C = 2
For cued modality: e A , C = 2 ( t ) = feedback A ( t ) − s ^ A , C = 2 ′ e_{A,C=2}(t) = \text{feedback}_A(t) - \hat{s}'_{A,C=2} e A , C = 2 ( t ) = feedback A ( t ) − s ^ A , C = 2 ′
For uncued modality, it can compare its measurement to its estimate like unsupervised recalibration (Zaidel et al. 2013):
e V , C = 2 ( t ) = s ^ V , C = 2 ′ − m V ′ e_{V,C=2}(t) = \hat{s}'_{V,C=2} - m'_V e V , C = 2 ( t ) = s ^ V , C = 2 ′ − m V ′
Model averaging:
e A i ( t ) = P ( C = 1 ∣ ⋅ ) ⋅ e C = 1 ( t ) + ( 1 − P ( C = 1 ∣ ⋅ ) ) ⋅ e A i , C = 2 ( t ) e V i ( t ) = P ( C = 1 ∣ ⋅ ) ⋅ e C = 1 ( t ) + ( 1 − P ( C = 1 ∣ ⋅ ) ) ⋅ e V i , C = 2 ( t ) \begin{aligned}
e_{A_i}(t) &= P(C\!=\!1 \mid \cdot) \cdot e_{C=1}(t) + (1 - P(C\!=\!1 \mid \cdot)) \cdot e_{A_i,C=2}(t) \\[6pt]
e_{V_i}(t) &= P(C\!=\!1 \mid \cdot) \cdot e_{C=1}(t) + (1 - P(C\!=\!1 \mid \cdot)) \cdot e_{V_i,C=2}(t)
\end{aligned} e A i ( t ) e V i ( t ) = P ( C = 1 ∣ ⋅ ) ⋅ e C = 1 ( t ) + ( 1 − P ( C = 1 ∣ ⋅ )) ⋅ e A i , C = 2 ( t ) = P ( C = 1 ∣ ⋅ ) ⋅ e C = 1 ( t ) + ( 1 − P ( C = 1 ∣ ⋅ )) ⋅ e V i , C = 2 ( t )