Cog Av Hearing - Project Vision
We welcome your ideas and thoughts, and would love to hear from you. Please contact us via the details on
the contact page.
Research Hypothesis and Objectives
Our hypothesis is that it is possible to combine visual and acoustic input to produce a multimodal hearing
device that is able to significantly boost speech intelligibility in the everyday listening environments in
which audio-only hearing aids prove ineffective. To test this we aim to develop and clinically validate a
next-generation cognitively-inspired, AV hearing device. We will achieve this aim by combining contrasting
approaches to speech enhancement developed respectively at Stirling and Sheffield in a novel AV enhancement
framework. Five objectives will be met in the process:
To combine signal processing from the enhancement framework pioneered at Stirling with scene
analysis models developed at Sheffield to produce perceptually meaningful acoustic features suitable
as input to AV enhancement algorithms.
To further develop and evaluate novel approaches to visual tracking and feature extraction in the
context of the AV enhancement framework. These approaches will be built on the 'bar-code' model of
human facial feature processing developed at Stirling.
To integrate two different approaches to enhancement, namely noise-filtering (Stirling) and
speech-resynthesis (Sheffield) in a common AV framework that takes advantage of their complementary
strengths. Integration will be considered at multiple (coarse and fine) scales.
To design intelligent multi-modality selection mechanisms that weight AV input and select the most
appropriate enhancement mechanism matched to environmental conditions.
Finally, to evaluate and optimise a real-time software prototype using a new AV corpus based on real
speech-in-noise scenarios. The prototype will be clinically evaluated using speech quality and
intelligibility tests with hearing-impaired volunteers.